Does DuckDuckGo offer an internship

European search index - sustainable to the Google alternative

Trapped in the filter bubble

Most people - including me - find themselves in their own digital filter bubble every day. Algorithms determine which information is relevant to us. You design the arrangement of the information in our timelines, feeds or search engine results. If you spin the thought further, I ask myself the following questions: On what information basis should we then exchange ideas? How should discourses - without which democracy is not possible - be conducted when everyone refers to information that has been tailored to him or her?

Where do search engines get their answers from?

After all these thoughts went through my head, I started to rethink my behavior on the Internet. After all, there are already many opportunities for us to regain a piece of autonomy on the Internet - but it is not enough. After changing my messenger and using open instead of proprietary software wherever it is currently possible, the monopoly-like search engine - you know which one I mean - should go to the collar. Unfortunately, this is not as easy as I had hoped, because there are actually no alternatives - many are now probably thinking "Of course there are alternatives, for example Startpage or Duckduckgo". Unfortunately, I can only soberly state that these are not “real” alternatives at all. In the following I explain why this is so. So far, I haven't given much thought to where search engines get their information from, but that's where the problem lies.
I started looking for an alternative to Google and came across an effective way to reinvigorate competition in the search engine market so that users can choose which search engine they want to use.

A valuable treasure trove of data

In the past, libraries like those in Alexandria were the repository of knowledge accessed by a few people. Nowadays most people draw their knowledge from Google's search engine, which has a market share of more than 90 percent in the European search engine market. But where does Google's search engine get its knowledge from? Search engines usually access at least one of the four major global search indices. A search index describes the “data treasure”, ie the data stock without which a search engine would be worthless. It can be understood as an image of the web, because it contains all web pages that can be found. In addition to Google (USA) and Bing (USA), Yandex (Russia) and Baidu (China) are among the great "treasurers" who have their own search index.

Web crawlers index the web

Search indexes use so-called web crawlers to continuously expand their data collection. This is software with which web pages are cataloged by the crawlers "rummaging through" web pages, following the links contained therein and indexing the content found. So a search index is very valuable. Without it, it is not possible to build a search engine.

"World crawlers" index the real world

For a long time, data for the index of search engines has not only been mined by plowing through the Internet: Search indices buy data, e.g. movement data from mobile phones, health data, flight data, travel timetables, content of media offers behind paywalls, current map material and so on. This data must also be made available so that a search engine can function with the usual comfort.

Search engines need search indexes

One would think that we can freely choose from which search engine we obtain our information. In fact, in addition to Google, there are also alternative search engines such as Startpage, MetaGer, YaCi, Ecosia or DuckDuckGo. But even if we access one of the supposed alternatives, there is usually no way around the large search indices from Google, Bing, Yandex or Baidu. The search engines are usually based on at least one of the four major search indices mentioned above. Startpage, for example, is allowed to share the Google index, Ecosia is based on the Bing index, and DuckDuckGo also accesses Yandex and Bing, among others. The reason for this is that it is almost impossible for a small company to set up its own index, because it is extremely expensive to set up and requires a high level of technical know-how. The existing search indices therefore have a lead that individual small start-ups cannot catch up. That is not fair and inhibits innovation and diversity in the “search engine world”.

Search engines are instruments of power

Google dominates the search engine market in Europe and causes dependencies and economic damage. Among other things, the company abused its dominant position by introducing anti-competitive contractual restrictions on third party websites. As a result, the European Commission imposed the third billion-dollar fine on Google in March 2019.

Everyone wants to be on top

Google tries to take an indispensable position in our lives based on the data it has collected about us. Most people only look at the first page of search results. Accordingly, the rankings on the first page of the search engine are in great demand. Google has created a money machine in which mainly advertisers compete for the top rankings in the search engine. In addition, politicians, entrepreneurs and other public figures are afraid that their clean slate will be stained by negative headlines in the first search results. This shows that just the way Google arranges search results is an instrument of immense value.

The search slot is watching us

On the other hand are the users. Google has made it its goal to individualize and perfect the display of search results. However, “perfecting” takes place behind our backs. In the first place, Google would like to present us with the answer that seems perfect to us personally. The company is already doing this quite well, as it brings together all the data it collects and creates profiles about us. This is a great danger because search engines are part of the digital infrastructure. Google holds a monopoly position and tries to cement this by constantly expanding its data collection. It even advertises “to organize the world's information and make it generally usable and accessible”. That is a questionable claim to power, because in order to make all the information in the world usable, the company must first acquire all the information in the world. Google received the Big Brother Award in 2013 for good reason. For example, Eric Schmidt, who was Executive Chairman of Google until 2015, believes “most people don't want Google to answer their questions. They want Google to tell them what to do next. ”Larry Page, who developed the Google search engine with Sergey Brin, even claims,“ Search will be built into the brain. Eventually you will have an implant and if you think about something it will tell you the answer. “This is going too far! Because it is no longer about personal leeway that is negotiable. It is about non-negotiable fundamental rights. It's about the common good and democracy. Our digital infrastructure should not depend on dominant private companies (and some outside the EU). They make themselves so indispensable and can determine over our heads how information should be dealt with. In a way, we are the “gold donkey” in Google's business model and our data is the gold. The study “Illuminated, analyzed and sorted” by Sarah Spiekermann and Wolfie Christl makes this very clear.

Filters and censorship

While the danger with Google and Bing may seem a bit more subtle, Yandex in Russia and Baidu in China make it clear what kind of sinister dangers still emanate from search engines. According to political ideas and wishes, content can be filtered, censored and thus made inaccessible to citizens. In China, for example, the Tianmen massacre, during which student protests were violently suppressed, cannot be found in the Baidu search engine. Due to a lack of transparency regarding their algorithms, search engines are particularly vulnerable to political and economic manipulation. They can be misused as an instrument of “opinion making”, which is of course an enormous danger for democracy and freedom of expression.

Ranking algorithms can be manipulated

The way in which ranking algorithms work (how they "determine" the order of the search results) is kept secret by the search engine providers. On the one hand, this is a comprehensible step, because if the algorithms were transparent and easy to understand, virtually everyone would have the opportunity to manipulate the presentation of results by search engines. On the other hand, due to the non-transparent algorithms, it is difficult to understand whether search results are free from manipulation. In principle, this means that transparent algorithms facilitate manipulation from “outside”, while intransparent algorithms enable manipulation from “inside”, for example by the search engine provider itself, and by SEO spammers (Wikipedia). SEO spamming tries to influence the evaluation algorithms - via entries in one's own or third-party websites. In order to guarantee equal opportunities in the presentation of results by search engines and to avoid manipulation, a neutral institution - for example a European foundation - could check the way in which results are arranged.

Depending on foreign search indexes

If we want to get information in Europe, we are largely dependent on the existing search engines and thus also on the search indices from the USA, Russia and China. That's a problem! In extreme cases of crisis or conflict, it could even go so far that we lose access to the existing search indexes - and thus to important information and data. The information available on the Internet is currently being “managed” by the USA, Russia and China, as they have one or more search indices. Europe, on the other hand, does not have its own search index.

Respect European rules

The market location principle applies to the European General Data Protection Regulation (GDPR). This means that all companies that target EU citizens and want to do business in Europe must adhere to the GDPR. However, many foreign companies are subject to their own laws. This poses a problem because in the US, for example, companies can be forced to release data to secret services based on the Cloud Act (Wikipedia) and the FISA Act (Wikipedia). In this way, data from European citizens can also fall into the hands of foreign secret services.

Strong arguments for a European search index

A European search index would create the conditions for freely accessible knowledge that companies and private individuals alike can access. Freely accessible knowledge forms the basis for competition and innovation, but also for individual freedom of expression. The advantages of a search index for Europe are many. For example, investments in new products based on the index are indirectly encouraged. In addition, competition between search engine providers who can use the index would be stimulated.

There are many reasons for an open European search index:

  1. Europe would no longer have to rely on indices from the USA, Russia and China - so sovereignty is strengthened.

  2. European laws and values ​​form the framework for the search index and thus also for search engines and other products that are based on it. Diversity of search engines is conducive to democracy worldwide.

  3. There is the possibility for new business models of search engines that are not “pseudo-free in exchange for user data”. Users could, for example, finance search engines of their choice - by subscription, as a cooperative or through donations.

  4. The EU can ensure that the index can be used on fair terms and that there is healthy competition.

  5. We currently rely on individual search engines, which do not allow us any insight into the structure of their search algorithms and data collections. This enables manipulation and abuse.

  6. Access to data, information and knowledge can currently be restricted according to the economic or political interests of the respective provider.

Start building a free search index

The German Advisory Council on Global Change has mentioned the idea of ​​a European search index in an EU policy paper (p. 19, p. 22, p. 35) and in a report (p. 234) addressed to the federal government advocates the development of a public search index. The paper and the report also included suggestions and tips from digital courage, which has long been concerned with a European search index against monopolies. The Federal Government and the EU should tackle these specific demands now.

Technical expertise and large financial investments are the premises for building a European search index. The index should, if possible, include all content on the web and allow everyone to access it. Furthermore, it should ideally be financed by the EU, but organized separately from the state and independently in order to avoid unilateral political influence. The terms and conditions should also be designed accordingly. The construction of an open search index therefore represents an infrastructure task that ensures the separation of publicly financed infrastructure and privately operated services. This infrastructure can be compared with power grids or roads, which can basically be used for private purposes, but also for business purposes. However, the responsibility to create safe and fair access remains the responsibility of the state or the European states.

Don't give up before we start!

Even if the setup requires a lot of computing capacity as well as high financial and personnel investments: It is worth it! Economy and competition, but also digital sovereignty and democracy can be strengthened by a European search index. In addition, many innovative programmers and creative entrepreneurs would have the opportunity to build a variety of new products on the basis of a European search index. The profits generated on the basis of the index would then no longer only flow to a dominant provider outside the EU, but could benefit many smaller companies in the area.

Many of you might think: "It's too late anyway, our data is distributed everywhere" or "Google is easy to use and we can't compete with such a huge company". Perhaps you are right, perhaps I am just naive or idealistic, but even if so, we should not just sit and watch as our autonomy is being taken away from us, bit by bit. It is not nice to lose, but it is even worse not to start and accept defeat in the first place. In addition, it is not as hopeless as it may seem to many. The EU Commissioner for Competition received a lot of encouragement at the SXSW technology conference in Austin when it came to the question of how big tech companies can be tamed - by authorities - in favor of democracy, freedom and autonomy. The subject of data protection is also gaining relevance in the USA and measures to contain the digital corporations are being discussed. The digital world is constantly changing - and we can help shape it.

So I ask you:

Let us convey to politicians that we want to rely on an infrastructure that is accessible to all when searching for knowledge. It should not be administered by a few dominant companies and designed in favor of profit maximization and the exercise of power by a few. We need an infrastructure that enables a large number of products, including (special) search engines, but also scientific tools, to build on it. The search engines based on this should guarantee the protection of our data and not just reduce us to a consumer role in order to develop profitable models of our purchasing behavior. Companies from Europe who feel connected to our values, adhere to the rules and laws applicable in Europe and for whom data protection is a concern, should be able to build business models on the index and make them profitable.In any case, we should not allow human knowledge to be administered by a few oligopoly-like private companies that endanger innovation and autonomy through opaque machinations. In addition, we should not let our ability to think for ourselves be undermined by a company that wants to decide for us which information is important to us and which is unimportant.

The implementation of the search index requires the political will, mandate and financial support of the EU, because the project cannot and must not be carried out by individual companies. In its major report on digitization (p. 234) and in a policy paper (p. 19, p. 22, p. 35), which are addressed to the federal government, the WBGU (Scientific Advisory Board of the Federal Government on Global Environmental Changes) has supported the Development of a European search index pronounced. The intellectual foundation has already been laid. Tell friends, family and acquaintances about the idea of ​​the European search index so that the topic gains attention and the policy begins to implement it. Europe must develop an independent digital and free ecosystem that benefits everyone! I have to rely on that, because I cannot walk the path away from the dominant search engine Google on my own.