https://arstechnica.com/gadgets/2024/01/google-search-is-losing-the-fight-with-seo-spam-study-says/
Study finds "search engines seem to lose the cat-and-mouse game that is SEO spam."
I guess we all kinda knew that, but it's always nice to have a study backing your opinions.
What fight? Google is making money, and nearly everyone is playing Google's game following their tune. Google is definitely not losing.
A lot of people dont remember pre-google these days.
Normal search engines worked, but Google was better results.
Now that every website is gaming SEO and the top half of search results is ads that pay to be first...
Google isn't that much better. I went to DuckDuckGo recently. The only thing Google does better is local results. But that's because Google always knows where I am and where I've been.
There's no longer a reason to use Google as a search engine, except habit.
Pretty much same with chrome
The main thing that got me switching to Google back then wasn't the better results, but their promise not to collect or use our data.
That all changed after 9/11, but by then Google had grown so huge it was hard to avoid them.
Even so, I still went back to Webcrawler and the others quite a lot and never really consistently used one search engine faithfully.
I remember pre-Google. There were a few human curated sites back then (like DMoz and Yahoo). I'm thinking that might be a way to combat spam and AI sites. As a side bonus, maybe it will help de-Google the planet.
I'm looking for a Wikipedia-but-for-the-web, where human curators find real web content for me. I found Curlie.org, and tried to sign up for it, but never got a response back on my sign-ups. Still I'm hopeful for something like that.
Yahoo was DMOZ (its directory used DMOZ data).
DMOZ had 100k volunteers curating the content at some point, and had a whole complex process to prevent abuse and so on. It will be hard to get going again.
But yeah, who would've thought that a mere decade after being discontinued it would become relevant again.
DDG uses Bing as the search API, and I don't see any evidence that it doesn't use SEO as well.
Just to be clear; “SEO” or “Search Engine Optimization” is a technique marketers use to craft web pages in a way that tricks search engine crawlers into considering them more relevant. It is not something search engines themselves do, and in many cases they actively fight against it.
So, it’s not whether or not DuckDuckGo uses SEO, it’s whether or not they’re susceptible to it.
To add to that, Google is the big one.
So everyone tries to get around googles SEO prevention measures.the little guys just have to do literally anything different
Coincidentally, I happen to have been reading into SEO more in depth this week. Specifically official SEO docs by google:
https://developers.google.com/search/docs/fundamentals/seo-starter-guide
To be clear, SEO isn't about tricking search engines per se. First and foremost it's about optimizing a given website so that the crawling and indexing of the website's content is working well.
It's just that various websites have tried various "tricks" over time to mislead the crawling, indexing and ultimately the search engine ranking, just so their website comes up higher and more often than it should based on its content's quality and relevancy.
Tricks like:
Those docs linked above (that link is just part of much more docs) even mention many of those "tricks" and explicitely advise against them, as it will cause websites to be penalized in their ranking.
Well, at least that's what the docs say. In the end it's an "arms race" between search engines and trickery using websites.
There's no longer a reason to use Google as a search engine, except habit.
I need to rollback to Google from DDG because the latter seems to refuse to understand that I want to find specific words with ""
And DDG isn't perfect either, I need to add Reddit as well more than I'd like to.
The Google ads team is functionally all of the company's revenue.
Google search still remains their most used product offering with most of their ad revenue (58.1% in 2022).
Google leadership is terrified that anyone could eat their lunch, because they know the search offering is getting worse and worse.
The origin of Google was taking out complacent search companies that had gotten comfy.
I'm pretty sure when I was laid off (1 year ago yesterday ❤️❤️ thanks Google) it was because they saw LLMs as a threat they hadn't taken seriously enough... Combined with that asshole billionaire being pissy that Google was only making 1.2 million per employee instead of 1.3 million.
With the end result of enshittification, people will migrate if their experience is bad enough. Google wants to strike a balance between making as much money as humanly possible and making the search experience at least decent enough to retain the majority of their users.
I would venture a guess that most people aren’t even realizing that their results are crap. I can’t even see them realizing it until after, I don’t know, all of the products they found via Google search and purchased wound up being gimmicky crap like MyPillow? Even then, I would be really surprised if they figured out what was going on.
True, if you look at YouTube, it's been getting worse and worse over time and yet people still go there, but that's also due to there being not that many good alternatives that have a bunch of content. Google has a ton of other good alternatives to compete with, so they're betting on the laziness factor and probably that people don't know better.
I switched to DDG merely to get rid of Google's irrelevant paid results up top.
If I'm searching for brand model manual I don't need every competitor's marketing detritus.
Likewise contact details etc... it's maddening.
Yes, but DDG seems to have worse organic results in my experience. I mean the bar is low, but DDG falls under it.
Oh yeah, the results are worse, but at least I can filter with search grammar and not also have to mentally filter out the ads too.
I'm not sure if we've ended up back at the ol' altavista/askjeeves keyword-stuffed hell yet, but it's close.
IMHO, the problem with Google isn't SEO. It's Google. When Google was great, it would find exactly what you were searching for. The whole point was to get you off of Google and on to whatever site you were looking for as quickly as possible. Over the last several years, their search has increasingly been drinking the 'engagement algorithm' Kool-Aid. Now Google doesn't search for what you ask, it searches for what it thinks you are trying to find. Which is fucking useless because I know exactly what I'm trying to find and that's exactly what I typed in. Selecting verbatim search and putting things in quotes helps. But it's still displays tons of irrelevant stuff that doesn't include what I searched for.
It's actually easy to point to exactly when the downfall started. Years ago Google was trying to make a social network called Google+ that would compete with Facebook. Before this, a + operator in the search field meant only show results that contain that particular term. But they wanted people to search for Google+, so they changed it so the plus sign became a searchable term and quotes were necessary to include a term or phrase. That was the moment Google decided that search wasn't their most important product. And it's been slow downhill ever since.
Okay, sure that was bad. But consider all the value that we've gained by having a lively and competitive alternative to Facebook! I mean, who do you know that doesn't treat Google+ as their first point of contact with the internet?
Lol Don't know anybody that does that, not since they closed in 2019 :P Amusingly, double quotes are still the standard 'must include' operator on Google search.
Google has also completely blown a very good opportunity to make a ubiquitous chat system. Several iterations of Google talk and Google meet and the like, only one of which federated outside of Google, none of which are compatible with each other, all of which seem to get remade or rebranded every few years.
Competitor to Facebook would have been a great idea. I had actually planned to join Google+. But shortly after it launched they started pushing it so fucking hard, like almost sneakly signing up people for it and making it damn near required to do anything, that made me say hell no. I'm pretty sure I wasn't alone in that regard.
I don't know what the hell is going on at Big G HQ, but it doesn't seem like they have much of any real mission these days. Haven't really since 'don't be evil' stopped being part of their mission statement.
I don’t know what the hell is going on at Big G HQ, but it doesn’t seem like they have much of any real mission these days.
The company is increasingly compelled by Wall Street pressures
The latest round of layoffs was practically dictated by activist investors like Christopher Hohn.
Stupid short sighted crap too. Complaining about excessive compensation and too much stock given away... That's the people who build the best generation of money making products there. If they have no skin in the game and aren't being compensated well, they aren't going to attract and keep the best talent. The best talent is going to go to companies like Tesla and OpenAI and various startups where those people have a chance to become millionaires on stock options.
It's one thing to pull the Netflix strategy, keep only the very best of the best people, pay them a lot, and get rid of everybody else. But treating labor overall like a cost and not an investment is not a good long-term strategy.
If they have no skin in the game and aren’t being compensated well, they aren’t going to attract and keep the best talent.
I think the theory being challenged is that "best talent" translates to "most lucrative product".
Certainly, there's no shortage of shitty mass market crap that makes enormous amounts of money purely by saturating the market. What's more, the model of cornering the market through regulatory capture or cartelization means that the talent of your staff has less and less of an impact on your market dominance. Eventually, when you've got a full blown monopoly, the only thing you really care about is the margin on your sales.
It’s one thing to pull the Netflix strategy, keep only the very best of the best people, pay them a lot, and get rid of everybody else.
Netflix hasn't even been following the Netflix strategy. They're routinely cutting bait on the highest watched shows and opting for cheaper productions with less overhead. One reason they love pumping out anime stems from the fact that licensing an English sub of a foreign media import is crazy lucrative relative to sourcing original content from Hollywood.
WB is taking this strategy into overdrive with their quest to saturate HBO with reality TV and old movies.
Generally agree.
But it depends on what your product line is. Does Google want to be Microsoft (new flavor of the same old crap, cloud centric, no special talent needed just competent coders and project leadership) or do they want to be OpenAI (push the envelope of what's possible and commercialize it)?
If the goal is to be Microsoft, this investor's comments are accurate. Fewer staff, less compensation, just get a few well paid product managers with a vision and a buggy whip to drive the coders to build it. Higher margins will mean more profits.
If the goal is to be OpenAI, this investor is dead wrong.
Netflix hasn’t even been following the Netflix strategy.
I was talking about their employees, not their content. Their content strategy is brain dead. They cancel so much stuff that it's not even worth getting into a Netflix show because it'll probably be cancelled after one season.
It might work, in the short term. But without quality content people will give up on Netflix and they will be the 'budget option'.
HBO is doing the same thing- I really think their management must be on drugs or brain damaged or something. HBO was THE most recognizable brand name for QUALITY content in the entire industry, and they killed it in favor of 'max' which is generic and means nothing and blends in with everyone else's 'plus'. And lopping off their own content is equally stupid.
Just like Boeing putting the useless McDonnell Douglas bean counters in charge of the company post merge, WB put the people who ran Discovery into the ground in charge. Sad to watch.
They cancel so much stuff that it’s not even worth getting into a Netflix show because it’ll probably be cancelled after one season.
Yeah. There was definitely some business goon looking at a spreadsheet and saying "Most shows get the max audience inside the first three seasons, so we should just cancel everything inside the first three seasons" without really considering what that means for the business model long term.
HBO was THE most recognizable brand name for QUALITY content in the entire industry, and they killed it in favor of ‘max’
Should be noted that they got bought out by the Discovery Channel precisely because the Discovery brand schlock was able to churn enormous profits relative to Warner.
Just like Boeing putting the useless McDonnell Douglas bean counters in charge of the company post merge, WB put the people who ran Discovery into the ground in charge. Sad to watch.
Boeing is testing the bounds of "too big to fail".
One thing about Max is that I don't really pay for it. I just get the service gratis through AT&T. I have to wonder how much of their business model effective boils down to "since people just subscribe and forget we can charge them indefinitely for schlook they'd normally click past on terrestrial TV".
I think this is the real wage of modern Streaming. An entire business model built on elderly people who never cancel anything.
My problem is replacing search terms with synonyms (which are often wrong for the context) and ignoring things like quotation marks or other search tools. It’s hard to exclude irrelevant results. Sometimes I’ll know an article’s exact title, search with and without quotes, and never find it.
It's time for Google to die. They are a truly awful company now so it's time to take her down to the shed like ol' blockbuster
I don't fathom paying to have your search history catalogued in correlation to your payment info. This will end as it always does, either hacked or enshittified.
The fundamental difference is that Kagi is making money from having the better product, not from serving more/better ads.
Kagi has started using search results from Brave's search index. The LGBT community disapproved of this because of past homophobic actions by Brave's CEO Brendan Eich.
Oh, that. Yeah, I'm not personally worried that they used it very lightly as one of a dozen sources and then stopped.
Fool me once...
Twitter has a similar problem. The more the CEO injects personal politics into the function of the site, the less confidence people have that a new search won't be fucked with. Whatever you might say about Google, Bing, and Yahoo, their owners have at least kept their politics closer to the chest.
Thats a terrible reason to not use something that works well though. I mean the founder or CEO of any major bank is probably a shit person with bad takes like racism but does it make their banking service any less useful?
No joke, I’ve been using Bing’s GPT-4 search and it’s helped me much more frequently than Google lately. AI might actually be where Bing out-competes Google.
How are you supposed to self-host a web crawler and indexer without getting a giant server bill?
Having this service at least slightly centralised makes sense ressource-wise - but assuming crawling and indexing is free is just foolish. I'd choose something like kagi but I guess many people will rather cheap out and go for the next free service not realising that that company has to make money another way to make up for the high cost of running a search engine
The Internet was tiny in 1998 but so were Google's servers. A little searching seems to show they ran everything on a dozen Pentium PC's with at total of 100GB of drives. That's less power than a single Raspberry Pi today with a $30 SD memory card.
I’d choose something like kagi but I guess many people will rather cheap out
I often feel as though these paid-for services aren't delivering a meaningfully better product. After all, it isn't as though Google's problem is that they don't have enough cash to spend on optimization. The problem is that they're a profit-motivated firm fixated on minimizing their cost and maximizing their revenue. Kagi has far less money to optimize than Google and the same profit-chasing incentives.
If there was a Github / Linux distro equivalent to a modern search engine - or even a Wikipedia-style curated collaborative effort - I'd be happy to kick in for that (like I donate to these projects). For all Wiki gets shit on ask Spook-o-pedia, they do at least have a public change history and an engaged community of participants. If Kagi is just going to kick me back the same Wiki article at a higher point in the return list than Google, why get their premium service when I can just donate to Wiki and search there directly?
If I'm just getting a feed of paywalled news journals like the NYT or WaPo, its the same question? Why not just pay them directly and use their internal search?
Other than screening out the crap that Google or Bing vomit up, what is the value-add of Kagi? And why shouldn't I expect to see the same shit-creep in Kagi that I've seen in Google or Bing over the last decade? Because I'm paying them? Fuck, I subscribe to Google and Amazon services, and they haven't gotten any better.
The problem is that it's just incredibly expensive to keep scanning and indexing the web over and over in a way that makes it possible to search within seconds.
And the problem with search engines is that you can't make the algorithm completely open source since that would make it too easy to manipulate the results with SEO which is exactly what's destroying google
you can’t make the algorithm completely open source since that would make it too easy to manipulate
I don't think "security through obscurity" has ever been an effective precautionary measure. SEO optimization works today because it is possible to intuit the function of the algorithms without ever seeing the interior code.
Knowing the interior of the code gives black hats a chance to manipulate the algorithm, but it also gives white hats the chance to advise alternative optimization strategies. Again, consider an algorithm that biases itself to websites without ads. The means by which you game the system would be contrary to the incentives for click-bait. What's more, search engines and ad-blockers would now have a common cause, which would have their own knock-on effects.
But this would mean moving towards an internet model that was more friendly to open-sourced, collaboratively managed, and not-for-profit content. That's not something companies like Google and Microsoft want to encourage. And that's the real barrier to such an implementation.
It's not about security through obscurity but "if a measurement becomes a goal then it ceases to be a good measurement" - so keeping the measurements hidden in order to make it harder for them to become a goal is a decent way to go on about it.
How would you measure "without ads"? That would just be the same cat and mouse game that adblockers have to deal with for decades.
I'm not sure it's possible to find a good completely open source solution that's not either giving bad results by down rating good results for the wrong reasons or that's open to misuse by SEO.
That might work if it's a small project where noone cares about fixing the results but if something like that becomes mainstream it's going to happen
Before we had Google, we had Altavista and before that we had indexes like Yahoo. Maybe we should consider going back. With the help of AI (I know...) it seems feasible to keep up with the ever growing content.
Maybe we should consider going back.
You can't really go back. Those old engines worked on more naive algorithms against a significantly smaller pool of websites.
The more modern iteration of Altavista/AOL/Yahoo has been the aggregation sites like Reddit, where people still post and interact with the site to establish relevancy. Even that's been enshittified, but its a far better source than some basic web crawler that just scans website text and metadata for the word "Horse" and returns a big listical of results based on a hash weighted by number of link-backs.
That system was gamed decades ago and is almost trivial to undermine in the modern moment. Nevermind how hard you'd need to work to recreate the original baseline hash tables that these old engines built up over their own decades of operation.
I need a better programming specific search engine. DuckDuckGo seems like it's gotten worse at code/project searches and will now just assume you misspelled some common word.
I tried Kagi last fall because everyone was raving about it.
Now I’m paying $5/month for search and couldn’t be happier 😅
What is Kati and is it actually good?
Edit:
Is it kagi?
Seems like something I'd be into but I'm also not a fan of my search results being logged against me.
First time I've heard of them. I like the concept but 5$ a month for only 300 search makes me think that you'd still need another search engine on the side, or pay for the more expensive plan.
The 300 searches goes pretty far because you usually get the correct result the first time.
Also you can just use hashbangs to search directly from a site like imdb, letterboxd, goodreads etc so those don't count against the 300 either.
also not a fan of my search results being logged against me
What do you mean? That you have an account so your searches are "linked" to you?
Well luckily for you, they don't do that. It doesn't maintain a search history at all which has its pros and cons. The only reason you have to login to use it is to check your payment level to determine your feature access. It is nice that login also allows you to use the same settings for multiple devices. One of those settings I really like is hiding results from certain websites (e.g. pinterest).
You can never truly know about almost any online service, you kinda just have to take their word for it, do some research, and pick the option that best matches both the performance and philosophy you're looking for.
I think they mean the fact they count each search you do, and depending on your pricing plan, can run out of searches in a month..
I'm not worried about Kagi's privacy, the only thing you need to give them is an email address (which is kind of a no-brainer if they need to contact you).
You can pay with Bitcoin if you want more privacy.
And they don't even allow you to store your own search history, because they don't want to save it anywhere.
"Save My Search History Currently this option can not be turned on. Kagi does not save any searches by default. In the future we may add features that will utilize your search history and then we will allow you to enable this."
Do you find the limited number of searches enough? I’d do it if it were unlimited for $5, but not going to pay $10 for a search engine.
I do 20-30 searches a day at work, so I definitely need to upgrade (annually it's around 100$ so not bad at all).
Their Universal Summarizer is awesome too
The summariser works on Youtube videos too btw...
You can just do !sum and get a summary of what they're saying in seconds. Changed my life.
Works fine for me on the starter tier, it used to behind the early access tier (Ultimate?)
It's either money or your data. I prefer to pay with money. If enough people do, the price might lower (hopeium) or the competition might increase for the same service, creating better or cheaper services in the same space.
Exactly. If you're not paying, you're most likely the product being sold.
People always keep forgetting that Google (or Alphabet) is an ad company, that does tech stuff with its 20% time. Everything they do is geared towards delivering more ads more efficiently.
This is why I've paid for email for years to Fastmail referral link
Similarly I started paying Kagi for searches, mostly because their results are better than DDG, which I used for years.
And, like the people I know from infosec, my phone is from Apple. They're the only company who makes you pay through the nose for the hardware and because of that specifically do everything they can not to know anything about you. (One of the reasons why Siri sucks so bad btw)
Depends. I use search engines a lot as a programmer. I have over 600 so far. But the results have been good. I'm willing to pay for that.
I thought I'd burn through them but I haven't had an issue. I use DDG for low hanging fruit but I'm going to stop bothering.
I'm at 243 searches now and it resets in 8 days. Seems to be enough for me.
I do use bangs quite a bit, so if I need to find a movie I go !imdb the matrix and if I'm looking for a book I can go !gr rando splicer which saves on search credits.
You may have changed my life with kagi. It is amazing so far at giving the results I want, and categorizing stuff like discussions so you don't have to add "Reddit" at the end of every search to get decent results.
And you can easily block or boost sites in the search results: I've got all pinterest domains blocked, along with a bunch of "news" outlets that don't actually produce anything good in the world.
Similarly I have stuff like reddit, hackernews etc boosted in the results if they match.
I did the free trial for a while and only used Kagi for a while, still going strong with the $5 300 searches per month tier. I don't really need to search that much because the results are usually correct the first time :)
Yep. I'm 100% sold on it. Thank you very much kind human. It's crazy how used to Google I had become. Not thinking about features which would be helpful like simply raising/lowering a sites visibility. It even has done well for me when it comes to local searches! Truly a game changer.
I've been using bing chat as my search engine for work stuff. Usually gives me the vendor kb or blog I need.
Google’s relevance of search has gone extremely downhill in the last few years even before the surge of AI articles, so it’s no surprise the keyword-injected articles are all that’s winning now.
But holy shit does it piss me off how many of these first page results have literally incorrect information now too. Want to learn how to do something in software? See a release date? Find accurate information? Good fucking luck.
For all people complain about the decline in ad revenue, there's still clearly a strong incentive to get click-bait responses.
I honestly wonder if the solution isn't to fight the SEO wars forever, but to just cut this shit off at the root and screen out sites that host ads, period. Obviously, Google can't limit its search by screening out folks that use AdSense because... that's half their business model. But perhaps a search engine that does bias itself against sites that monetize click-throughs could dramatically improve results.
Yeah it's pretty wild how bad search results have been lately. The unfettered proliferation of AI bullshit on the internet is gonna have some really goddamn irritating impacts on just about everything I think.
I’ve been trying DuckDuckGo recently and already started seeing some better results. I’m not ready to change my default search engine yet, but if it keeps up I could see that happening sooner than later.
I recommend reading the actual paper this article is about. DDG is actually by far the worst by their measures. Google is 9% spam compared to 31% for DDG and 23% for Bing. That's a huge difference.
I would recommend trying a SearXNG instance if you haven't before. You can combine results from multiple sources. I use Google as my main source while also having access to the DDG-style !bangs.
My family was looking for a specific manual and couldn't find it on Google, the only search engine they know about. It was one of the first results on DDG.
I'm using DDG for a while now and I'm happy with it overall.
Been using DDG as my default for about 4 years now. It’s actually just gotten better as Google has gotten worse. I used to use the bangs a lot to check Google from DDG, but do that less and less.
Same on the default search changing, I'm very much used to Google Search and it's gonna take me a while before I switch. Though I've already managed to switch from Gmail and it's kinda refreshing, so this will be just another step.
I remember back in the day when the same shit was happening because websites would just put, like, an entire dictionary in tiny, hidden text somewhere at the bottom of the main page so it would have a greater chance of showing up no matter what you typed into the search engine.
It's pretty wild that they didn't think about people doing exactly the same thing with all the new methods they came up with to make searches "relevant" since that was one thing that made Google so much better than the competition back then.
Any system you make to push certain things over other things are going to be figured out and gamed.
that's the thing, in the past they had this problem try to solve it and fixed it, now they just cash out from ads campaigns and fuck you customer stop complaining and submit
Arent they trying to make their search worse so you have to search multiple times, showing you more ads?
The study found that ddg was worse than google though. But they only searched for products, so nk complete "test".
Ddg is still good for non product shit, hell sometimes if im looking up niche enough shit itll give me results with for example space battle forums.
I would think they'd make more money by sending you to sites that use Google ads.
The ads on Google's own pages are often fairly benign as far as internet ads go. I've certainly never had them tell me I'm the millionth visitor and try to reward me with the gift of an epileptic fit.
one of my least favorite tech innovations lately is that they expect you to want to type in a full ass sentence instead of just 'macbook display dimensions'
I've been thinking of switching, because Google sucks nowadays, though I'm not sure I really want to switch to another big corporation if I'm taking the effort of switching.
I'm really enjoying perplexity.ai most of the time you just get the answer you're looking for.
Google search has,become so bad I couldn't even get the number of my insurance company yesterday. There was nothing but spam on the first page. I am in the market for something new.
Same. Though according to the article, Google is still better than competition, so yeah, not that many alternatives.
and the study itself points out that Google has improved over the past year and is performing better than other search engines. More broadly, numerous third parties have measured search engine results for other types of queries and found Google to be of significantly higher quality than the rest.
I've been noticing this the past 6 months or so. It's not much different than duckduckgo now. And now I'm thinking I need to find more search engines since google is no longer the end all be all when I'm looking for something.
I would expect its only going to get worse as they try to suppress wages across the board in tech
