As an online publisher, you may or may not know that there are different types of search engine, infact you might be wondering how these search engine sites make money.

Some of them may be integrated into your site and some of them work on a stand-alone basis. From vertical search to sponsored directories, there are quite a few ways to integrate a search product into your business model.

• What is a Search Engine?
• Core Components of a Search Engine Monetization Approach
• Types of Search Engine
• How Does Search Engine Sites Earn Money
• How Do Private Search Engines Make Money?
• How to Create a Search Engine Site
• How Much Does it Cost to Develop a Search Engine

What is a Search Engine?

A search engine is web-based tool users utilize to find certain information on the Internet. Usually, it is an automated software application which can perform several functions:

• Crawling. Crawlers search on different websites at the same time to collect large amounts of information that enables the search engine to find up-to-date content.
• Indexing. After crawling, the search engine usually indexes the found content. It is based on the appearance of the keyword phrases on each website individually and allows fast and easy query and subject search.
• Storing information. To make the search quick and easy it is crucial to store the web content within the database.
• Giving Results. These are the hyperlinks to the websites that appear in the search engine after you have typed your query.

The commercial search engines are different from the public ones (Google, Yahoo, Bing, and others). Yeah, their crawlers somehow rank the content, but the entire process is a bit more complicated. The commercial search engines deal with vast and complex data processing. Their search algorithms are still looking for the patterns, which can describe the data unit, but they do it differently.

Core Components of a Search Engine Monetization Approach

Some of the core components of a search engine’s monetization approach include the following:

Relevancy before monetization

For some portals, search is their core business. For others, like Yahoo or AOL, it is part of a much broader business. To achieve maximum benefit out of any monetization strategy, a search engine company must keep the priority on relevancy (and diversity) of results; an updated, user-friendly interface; ever-increasing inventory; and improving the search tool’s understanding of the audience’s query type.

Provide unique search technologies

A search engine has to consider what differentiates its search product from others, and whether or not it clearly provides a perceived value to both search users and advertisers.

Read Also: How Google User Page Experience Can Drive Your Business Profit in 2021

One of AskJeeves’ challenges was how to differentiate itself from the most well-know search engine – Google – by doing something unique with its own search product that people find appealing. “What differentiates us is that our audience is used to giving broader type queries,” said Jim Diaz, SVP of Sales and Business Development for AskJeeves. “A substantial majority of people that come to our site will ask more than three words in a typical keyword query.

That’s substantially higher – even off the charts – compared to most search engines, where you typically get one or two queries.””We’re not trying to out-Google Google,” added Diaz. “Unless your product is different and offers something unique to the user, it’s going to be very difficult to make a name for yourself and stick around in the long-term.” 

Encourage commercial search 

Commercial search accounts for 30% of all search queries, according to Rashtchy’s own research. However, the search engines find these queries to be just as complex as their non-commercial counterparts. “Search users articulate wide range of commercial needs,” said Tim Cadogan, VP of Search for Yahoo Search. “That involves showing a decent quantity of relevant results, not just one or a couple. And for a significant portion of queries, users are expressing needs that businesses satisfy best.”

Present a diversity of vendors, partners, products, and sales channels

All search engine companies have created paid inclusion or paid placement programs to obtain broader reach for more targeted advertising audiences.

Offer affordable or value-oriented pricing models

While search engines make no bones about going after the big fish, monetization strategies rely heavily on expanding opportunities for the majority of businesses. “We also have the opportunity for small-to-medium advertisers to sign up and get ads on their pages,” stated Sandberg. “We’re already seeing people who had pages on the Web – that they were not able to monetize before – are now able to monetize and target for the first time.”

Motivate advertisers to improve content and conversions 

“There are always a lot more advertisers that want to show up for a commercial query than there is space available,” said Tony Mamone, Vice President of Small Business Services for LookSmart. “By charging for that content, you’re motivating that advertiser to improve on their own site.

They’re paying for that click, for the leads that are coming through. They’re not going to want all-you-can-eat; they’re going to want all that’s relevant. It also calls out the bottom performers over time. And for those who request help, you can provide them with a team to assist with optimizing their content and making sure it is relevant and more productive.”

Globalize your ad market

Obtain a broader reach geographically and provide localized services for each region. “More than half of Google’s 200 million searches each day are done internationally, outside of the US and Canada,” said Sandberg. “Similarly, our advertising program has also gone global.

At Google, you can target your ads for any country in the world. You can also pick your keywords in any language in the world. We even run our international advertising program with localized customer support in eleven languages and we take payment in six currencies.”

Keep operational expenses cost effective

Infrastructure to carry for so much search traffic can be very costly if not carefully supervised. Search engine companies must regular monitor peak usage times on their Web servers; improve on their software efficiency with Web servers; and limit the size and complexity of database sizes and complexity of the search nodes.

While many individual performance models were described, paid inclusion was a particular issue of contention between among audience members and the panelists.

The search engines touted paid inclusion being one of the most effective monetization strategies that successfully delivers more relevant results.

Yet some in the audience expressed concerns on how it might be affecting the relevancy of search results. “How can search results be relevant as Google’s results if paid inclusion is such as important part of each of your programs?” one audience member asked.

Types of Search Engine

Here are the five different types of search engines:

Crawler (Google): This crawler search engine can be stand-alone (like Google.com) or be embedded into your site (like Google Custom Search). A crawler search engine generates its results by the automatic compilation of of web pages and sites. In other words, it works with its own algorithm and delivers search results with no intervention.

Directory (Open Directory Project): A directory search engine is a user-generated collection of search results and sites. The Open Directory Project (dmoz.org) was constructed and is maintained by 81,586 volunteer editors. Its search results come from 4,581,316 sites in over 590,000 categories that were submitted by users across the web.

Vertical (Business.com): A vertical search engine can be stand-alone (like Business.com) or can be integrated into your site (like Pure Contemporary Magazine). The results from this type of search engine come from sites that are hand-selected by the publisher. Google Custom Search is an easy tool you can use to start a basic vertical search engine.

Sponsored (Kanoodle.com): A sponsored search engine relies on search results that are sponsored by a retailer or publisher. Buying your way into a sponsored search engine like Kanoodle also gets you into CNET’s Search.com, and the InfoSpace properties, including Mamma, WebCrawler and Dogpile.

These paid-inclusion results don’t normally come with a ranking guarantee; instead, they guarantee that the site gets indexed immediately instead of the normal schedule of a crawler search engine, which is monthly. This makes sponsored search engines ideal for new online businesses.

Hybrid (Yahoo!): Hybrid search engines rely on both user submissions and crawling. Yahoo! Directory Submit is a paid-inclusion model where a publisher or retailer pays $299.00 a year to get their site indexed within seven days to the Yahoo! Directory.

They also run Yahoo! Sponsored Search where companies can get listed as a sponsored result on search result pages. In addition, the results of user searches are all based on their own crawler algorithm (unless they’re sponsored).

How Does Search Engine Sites Earn Money

• Display Advertising: Charging sponsors for text and image ads in search results (CPM).
• Keyword Advertising: Charging websites for individual keywords that display relevant ads in search results (CPC).
• Paid Inclusion: Charging to be listed in a vertical or otherwise targeted search engine or directory.
• Habit Reports: Selling marketing data on consumer habits or using it for targeted advertising.
• Syndication: Charging other search engines to use their results catalog.
• Any/All: Any combination of the above.

Search engine like Google earn money from search related advertisements. If you search something in Google it shows keyword related ads at the top and on the right side. Every time someone clicks on one of these ads, the search engine earns money from pay-per-click basis.

Advertisers pay for placements in the search results for keyword phrases of their choice. The search engines tries to provide the best user experience with more related ads. The more related results, the more likely the user will click on an ad.

Search engines have the interest to maintain quality search results – the better the search results for a given query the more likely a user will return to use the same search engine again.

The more number of users a search engine has the larger the possible target audience there is for the paid advertisement and hence it increases the revenue for the search engine company.

How Do Private Search Engines Make Money?

In recent years, the supremacy of search engine giants like Google and Bing has begun to be challenged. Growing concerns over privacy violations have driven people to find safer alternatives in private engines like HotBot.com, DuckDuckGo, and Startpage.

Unlike Google, these search alternatives don’t use tracking cookies or collect user information at all, keeping your identity safe. However, much of Google’s massive income is generated through the use of targeted ads created by the data they collect. 

So how do private search engines, with no saved user information, create revenue in order to operate?

Contextual Ads

While private engines don’t use targeted ads to earn income, that doesn’t mean they don’t advertise at all. Instead, they use contextual ads. The difference between the two all comes down to tracking. If you were to look up socks on Google, Google would store the information that you’re interested in socks. If you later look up brownie recipes, Google might still show you ads for socks because it knows that’s what you were interested in before. That’s called a targeted ad.

Contextual ads are based on keywords. If you look up socks, you will be shown an ad for socks. If you look up brownie recipes, you will be shown an ad for brownie recipes. As soon as you navigate away from a page, the private engine forgets what you were looking for and any other ads you see will be directly related to the search you are making. This is how we at HotBot.com earn most of our revenue.

Private Search Engines Make Money Through Affiliate Links

Many private engines earn income through the use of affiliate partnerships. With this method, they partner with a company or organization interested in increasing traffic to their website. The partner provides them with a link which contains a specific ID or username. The affiliate partner is able to log how many times the link is clicked and pays a small amount of money to the search engine for every click, as a reward for the engine generating traffic.

Shopping Search

Some private search engines are beginning to integrate shopping features into their platforms. When you make a search, they might include a sidebar that shows you where you can buy products related to that search.

Every time a user clicks one of those products, the search engine earns a little bit of revenue. This method is very similar to the use of affiliate links but specifically directs to ecommerce stores, whereas an affiliate link can direct to any web page the partner wants traffic for.

Donations

Some private engines were designed as non-profit organizations and rely on donations for income instead of ads or affiliate programs. This is similar to how Wikipedia generates income to continue running.

Private search engines are still small but they’re also mighty, and growing every day.

How to Create a Search Engine Site

If you are planning to build your own search engine, there are certain rules to follow.

There are two stages of the process and each has several steps.

The First Stage

This stage helps you prepare for developing your own engine software as well as explains how to launch it successfully.

Step 1. Write down the search requirements

First, you need to write down the requirements for the search. To understand this, you have to answer the following questions:

• How much data is planned?
• How many searches will be there?
• How often will the data be updated?
• What features do you need?
• Is aggregation needed?

Step 2. Select an engine

The second step of making your own search engine is to choose the engine itself. Exactly, there is no need to build a search engine software from scratch, you can select the existing one and tune it according to your needs. They are also very well optimized in terms of efficiency.

Solr, Elastic Search, Sphinx, Xapian are ones of the most popular. Let’s have a closer look at them.

Elastic Search

It is open-source with refresh interval of 1 second which started in the early 2000s is financed by Elastic N.V. It helps customers explore and analyze different kinds of data like Apache logs and Twitter streams. It allows for the creation of the app, enterprise, and website search along with monitoring geo data, monitoring availability, and analyzing security events.

Solr

It is a dependable and scalable open-source enterprise search platform that provides load-balanced querying and replication, distributed indexing, automated failover, and recovery. It was created in 2004 with updates approximately every year. The last one took place in March 2019.

Sphinx

It is an open-source search service that provides such services as consulting, package matrix, embedding, enterprise support, etc. Indexing speed of Sphinx goes up to 10-15 MB/sec per core and HDD. First it was launched in 2001 with the last update in 2018.

Xapian

This search engine library is created to help developers add search facilities and advanced indexing to their applications. It has partly evolved from Open Muscat engine which was first designed back in the 1980s. Updates take place every year or so with the latest version presented in September 2019.

Step 3. Start the Engine

The next step is to start the selected engine. Setting the analyzers and compound queries along with arranging the boosts for the fields are the main processes that you have to do at this step.

If you prefer using Elastic Search, as we do, you can use their own service. It makes the process of deployment, security, and operation of the Elastic Search on a large scale easy and fast.

Step 4. Define Index Structure

When building a search engine software you have to determine the index structure. Even though it is a kind of database, it is important to remember that this is not the main data storage, neither it is a relational database. The index structure must be organized in a way that is convenient for the search. The data stored there also has to be the only one which is necessary for the search.

Step 5. Set Up Data Update

It is important to send the updated information from the database to the search engine. Some engines get this information directly from the database when in other cases you have to add a special code that completes this task. The search engine is more efficient when updates are rare. So, if there are dozens of queries per minute, it would be better to set the index update once per several minutes. This will allow sending numerous updates together.

Developers working with Elastic and using Python could utilize Github service and Celery to plan the index update.

Step 6. Start making requests

At this stage, your search engine works well and might not require any additional work. Therefore, you can start making requests.

You can use different ranking algorithms that apply the data on the frequency of the word in texts and the engine knows that the main word in the “cardiology services” query, for example, is cardiology. You can use different ranking algorithms that apply the data about the word frequency in texts.

So, in the phrase “cardiology services”, the engine can identify the word “cardiology” as the main one. Therefore, the results matching both words go first. Then, there will be the ones matching “cardiology” and the other ones matching “services”.

When working with Elastic, we prefer Elastic DSL. There are several reasons why:

• It is able to build index automatically which is very convenient at the prototyping stage.
• Its http-based api is user-friendly and allows for coding in any programming language.
• There are numerous instruments available such as Kibana and Logstash.
• Amazon offers Elastic as a service which simplifies the launch and administration of the search engine.

This is where the first stage of creating the search engine design comes to an end and the second one begins.

The Second Stage

This stage deals with other processes that help make your search engine more efficient.

Step 7. Assign a Responsible Person for Data Collection

First of all, you need to hire an expert who specializes in databases. Even though setting up a search is a technical task, a technical specialist may not be able to understand what kind of data users need and why. This is when a data specialist comes into use.

Step 8. View User Search History

It’s important to find out if the results of your search engine are suitable for certain queries. It can be done by checking the user search history, choosing the top ten queries according to the popularity and letting an expert check their relevance.

Step 9. Formulate What Documents Are Expected as a Result

Next, you have to formulate what documents are needed as a result. This is when you need to think about how you, as a human, would process such queries. For instance, you are working on scientific articles and as a result, you may get the following:

1. Matches in the name of the article are more important than matches within the text.
2. Matches within the text are more important than matches in the references.
3. Matches of the author’s name are more important than matches within the text and in the list of quotations.
4. Name and surname must be searched together, not separately.
5. The word “vaccine” is usually misspelled as “vacine” and this query must be processed as well.

Step 10. Find out the Source of the Problems

The final step is to find out why problems occur if there are such. Reading the information on how the engine search is built and the methods of its troubleshooting can be helpful. Sometimes you might need to readjust the basic principles to find the problem. However, sooner or later problems that require a debugging mode and detailed analysis will appear.

Depending on your search engine rules, you may need various ways to fix the query, which will always be interactive. So, identify the problems, sort them out and try to enjoy the process.

If you are working with Elastic, there are a few tips to help you make a search engine software for your business:

1. Read about all the analyzers. Usually, only two or three of them are used, but you need to know about the others.
2. Understand how compound queries work, especially the Bool query. You can find more information about it here.

Use appropriate weights and boosts. There is a great book “Relevant Search With Applications for Solr and Elasticsearch” by Doug Turnbull and John Berryman that might be helpful.

How to Hire Developers

To make your own search engine you need to hire specialists that meet your requirements. There are several alternatives. Let’s look at the pros and cons of each of them.

In-house Team

One of the options is to have an in-house team.

Pros:

• Such a team is usually more professional and more involved in the process.
• You have total control over the team’s work.

Cons:

• You have to pay quite a high cost to develop a search engine software for a team of skilled professionals.
• It may be difficult to find qualified developers.
• There is no guarantee they will work well in a team.

Freelancers

If you are not able to hire an in-house team, you may try to work with freelancers.

Pros:

• They charge far less money than an in-house team.

Cons:

• Finding experienced freelance developers is not easy.
• There are certain risks: they can suddenly disappear or not be able to meet the deadlines.
• You might have to hire a project manager to keep the process under control.
• Communication between the members of the project requires additional help.

Outsourcing to an Agency

The third way, which many companies successfully use nowadays, is outsourcing the task to build a search engine software to an agency such as Gearheart, for example.

Pros:

• Such agencies have skillful and professional employees that have a great experience in this field.
• You pay only for the actual time spent on the development.

Cons:

• Sometimes such agencies might not able to meet a deadline or provide a product of the expected quality.

You can avoid such problems as mentioned above by choosing an agency wisely — it should have a good reputation and the skills of the developers have to meet your needs (this can be checked in their portfolios). And, of course, you should never hesitate to ask questions whenever they occur.

How Much Does it Cost to Develop a Search Engine

From our point of view, the development cost is probably not the first thing customers should care about when calculating the final costs. There is another aspect that should be taken into consideration – maintenance.

If we look at the big brother – Google – we see that there are many servers (hundreds of thousands, probably) processing data in real time.

Why do they do so? The World Wide Web is a fast-changing environment. There are both static and dynamic pages, and all those pages should be recrawled multiple times to track the data changes (if there are any).

In this way, Google processes the same data over and over again to make SERP fit the user query. It is the best and the most effective way to monitor changes, mainly if there are sextillions of pages available.

Large search engine companies use complex algorithms to look for “footprints” in the document. For example, we do not need to collect all the data about the book, when we can spot the key thesis in its summary. This way, we recognize the above-mentioned footprint that contains the necessary data: author, titles, summary, brief description, keywords, publication data, etc. – and add this footprint to a separate database.

When the user instructs Google to find something, the search engine looks for the pattern in the footprint database first. If it doesn’t find a matching answer, it performs a deep search. If this is the case, the pages are generated at a slower pace. You can check it yourself – make a complex query and compare the SERP generation time for different pages (usually, the first page pops up far quicker than the thirtieth).

Read Also: How You Can Make Money Online With Google Adwords

If there are hundreds of thousands of servers needed to perform a search – so how much will it cost? Well, nobody knows the exact estimates. The only thing we know for sure – a lot. Google is now setting up new powerful servers to process data in a quicker, more accurate and secure way. Thus, even the most complicated and in-depth queries will be performed in an instant and generate precise results.

We discovered how Google works. Now let’s see how commercial search engines process data.

We can use two approaches:

– Develop a lightning-fast search engine powered by solid mathematical knowledge, modern databases, SSD drive and coded with the fast programming language like C++

– Develop a “footprint” database

These two approaches affect search engine development costs. Customers usually prefer the first one as it is more accurate but slightly more expensive.

If you want to build a search engine from scratch in Python or PHP, for example, you can do it for free after completing some courses at Udemy or EDX. It requires some programming skills though. In the case of paid courses, it will cost you up to $100.

If you want to create a search engine like Google (with a decent search quality), we would say it might cost you about $100M (for the prototype) – including costs for servers, bandwidth, colocation, electricity and so on. Maintenance costs for the existing cluster may go up to $25M per year.

If you want to create a commercial search engine for your business – be it the insurance, bio-informatics, healthcare, e-commerce, or other company – the development costs may range from $10,000 to $60,000, with a low maintenance fee.

Creation of a search engine software is a great way to enhance the opportunities of your business. What’s more, it can be interesting and fun if you follow certain rules and enjoy the process.

Engaging a professional team of developers working with a special agency is always an advantage for your project because the work is done by skilled developers. So, choose the approach that meets all your needs and set off for a developing journey.