There are two types of files in Drupal, Public and Private. Public files, like the files uploaded to your site are public files. These are files that available to anyone that knows the URL.
If you do not want to make a file public, make it private!
To configure private files, navigate to admin > config > file system or admin/config/media/file-system