An introduction to Open Source Intelligence (OSINT) Gathering

Welcome to the monthly Tiberium Attack Surface spotlight! Each month we will be deep diving on specific topics in this area and most importantly, it will be on what you want to see! (voting via Tiberium Linkedin page). You can jump straight in with a private demo by getting in touch here.

So without further ado, here is our intro to just what OSINT is, enjoy!

The rapid technological advancements have shifted the entire world into what is now known as the information age, the main characteristic of this new era is utilising internet technologies in all life aspects and across all industries. This huge transformation to digital society resulted in creating huge amounts of digital data scattered across the web, and most of this data is open to public consumption.

Open Source Intelligence (OSINT) is a sub-type of threat intelligence that includes human, signals, and geospatial intelligence, however, unlike the other three sub-types, OSINT intelligence is only gathered from free, public sources.

In this blog, we will introduce the term OSINT, and see how OSINT can be used in various situations by different actors to support intelligence needs.

Defining Open Source Intelligence

OSINT refers to all public information that can be accessed by anyone without violating any copyright or privacy law. OSINT resources are not merely limited to online data, for instance, paper magazine, books, TV and radio broadcast are all considered a part of OSINT sources, but the continual digitisation of society has resulted in making most OSINT data comes from online sources.

When conducting OSINT, you should not limit your search to the contents found via typical search engines such as Google, Bing, and Yahoo! These search engines are only searching the surface web which only constitutes %4 of general web contents, while the remaining web contents are buried deep in the bottom layers (deep and dark web) and need special arrangement to access.

OSINT types

OSINT sources can be categorised according to where it is found as follows:

  • Internet: This includes anything published publicly online such as discussion forums, social networking sites, blogs, internet messaging applications, digital files (images, videos, documents, PDF files) including their metadata, technical information of websites (e.g. IP address, WHOIS info, web servers, web, and email provider), government public databases (e.g. Vital, crime, court and tax records), dark web resources (e.g. Contents found on TOR and I2P anonymous networks, including leaked data).

 

  • Traditional media channels: Such as TV and radio broadcast, newspapers, magazines.

 

  • Academic publications and scientific journals: This includes everything published by the scientific community for free or can be accessed on a subscription basis.

 

  • Business papers: This includes everything related to enterprises’ works such as business profile, tax info, board meetings, annual records, tenders, import/export deals.

 

  • Geospatial information: such as Geo-location info generated from the Internet of Things (IoT) devices, commercial satellite images, and free online map data.

 

Who Needs OSINT?

Both individuals and enterprises can benefit from utilising OSINT data, the following mention the main beneficiary actors:

  • Law Enforcement: Law officers utilise OSINT for tackling all forms of crime in addition to increasing their intelligence collection capabilities to counter crimes and protect society, businesses, and public entities from cyber-attacks and other organised crime.
  • Intelligence agencies: Government agencies utilise OSINT for various purposes such as protecting national security, counter terrorism, cyber-tracking of terror networks and to predict future trends globally in different areas (e.g. Health, economic, agricultural and immigration …etc).
  • Business corporations: This category is considered among the huge consumers of OSINT sources. Enterprise needs OSINT to gain intelligence about its competitor’s activities, to plan entrance to new markets, or before launching a new product or service to measure public acceptance. In the cybersecurity field, OSINT assists enterprises in finding information about their own company, employees, and IT assets in addition to finding leaked data that can be exploited by cybercriminals at any time. OSINT also helps them to gain insight into threats coming from cyberspace. They monitor dark web communities for any mention of their business name, they research for any vulnerability or exploit that can threaten their IT system.
  • International and non-profit organizations: The UN and other humanitarian organizations utilize OSINT to plan their rescue operations in conflict zones and areas that suffered from natural disasters.
  • Individuals: Most internet users are using OSINT without consciously know it. For example, when you use Google to search for someone or to find a business address, then you are using some forms of OSINT to locate this info.
  • Ethical hackers and friendly penetration testers: Unlike criminals who penetrate IT systems to steal confidential information, ethical hackers use the same methods and tools to discover vulnerabilities in friendly networks, so it can be closed before exploiting it by other threat actors.
  • Black hat hackers and terrorist organizations: Unfortunately, OSINT can be used in the wrong way by criminals and terrorist groups to gather intelligence about their targets. Criminals search for vulnerabilities in target IT systems to find an entry point that can be exploited to achieve various malicious objectives.

OSINT challenges

One of the commonly overlooked areas of OSINT gathering for junior searchers is the risks and challenges associated with it. In this area, we can recognise the following:

  • Risk of revealing researcher identity: OSINT gathering should be an anonymous affair, revealing the searcher identity can heavily damage the investigative work and in some instances can have legal consequences.
  • The volatility of digital information: The volatile nature of web content imposes real challenges for OSINT gatherers, for instance, a suspect may erase incriminating information or deactivate his/her social networking account that contains important info.
  • The Sheer volume of data: The huge volume of OSINT data requires intensive human resources, automated tools can be utilised to filter results in manageable pieces.
  • Information trustworthy: The unreliability of data sources imposes another challenge, sometimes, an OSINT gatherer should verify a piece of information using more than one source or intersect it with other info gathered from a non-internet resource.

Prominent OSINT tools & techniques

Before we close things out, we will mention some popular OSINT tools/online services that can be used to gather and process OSINT information.

  1. The first technique that must be fully understood by any OSINT professional is utilising Google’s advanced search. As we already said, Google is only able to index the surface web content, however, even within the surface web, there is hidden information that requires using special search operators to find, let us clarify this with an example:

“sensitive but unclassified” filetype:pdf site:usa.gov

In this example, we are asking Google to filter the results and return only specific file type “PDF” from only one domain name “usa.gov”, and these files must contain the phrase “sensitive but unclassified” somewhere in the document text.

Google advanced search can be used to find sensitive information on the surface web

  1. OSINT tools directory: There are hundreds of tools, online services, and data sources that can be used to find OSINT intelligence, knowing about all these tools/services is a daunting task even for senior OSINT gatherers. Some websites offer a directory of links to OSINT resources, OSINT.link & www.osintframework.com are two popular options.
  2. BuiltWith: This online service reveals the type of technology used to build a specific website.
  3. SpySE: This is a specialised search engine for identifying internet assets, containing a wide range of OSINT data handy for the reconnaissance.
  4. Maltego: This is a data mining program used in computer forensics and open source intelligence (OSINT) gathering. Maltego can integrate with scores of online services to harvest data from a variety of public sources.

Summary

No matter what is your goals, OSINT has became an indispensable tool for all your security disciplines. Knowing how to utilise OSINT becomes the key to strengthen your cyber security defences and to increase your competitive advantages in an ever-changing information age.

This type of work is one of the things we are living and breathing here at Tiberium and we would love to hear from you if you would like to get in touch to learn more about it.

More from our blog