Sphinxsearch is a search engine for fast fulltextsearch, can get data from mysql, oracle and mssql, can act as a storage itself (realtime indexes). Also, sphinx has a mode of operation through api and through sphinxql - an analogue of the sql protocol (with some restrictions), which allows you to connect search through sphinx on the site with minimal code changes. This is one of the few great, large and open projects developed in Russia. In my life, I saw how sphinx processes about 100-200 search queries for 2 million records from mysql and at the same time the server breathed freely and did not feel sick, mysql starts to die already at 10 requests per second on a similar config.
The main problem of the sphinx documentation, in my opinion, is a small number of examples for most interesting settings, today I will try to tell about them in examples. The options that I will cover are mainly related to search algorithms and variations. Everyone who works closely with sphinx will not learn anything new, and newcomers will hopefully be able to improve the quality of search on their sites.
Sphinx contains two independent programs indexer and searchd. The first one builds indexes according to the data taken from the database, the second one searches for the built index. And now let's move on to the search settings in sphinx.
morphology
Allows you to set the morphology of words, I use only stemming. The stemming algorithm uses a set of language rules to truncate endings and suffixes. Stemming does not use ready-made word bases, but is based on certain truncation rules for the language, which makes it small and fast, but this also adds to its disadvantages as it can make mistakes.An example of word normalization by stemming in Russian.
The words “apple”, “apple”, “apple” will be truncated to “apple” and any search query with a variation of the word “apple” will also be normalized and will find records with the words that were described above.
For English, the words “dogs” and “dog” will be normalized to “dog”.
For example, in sphinx it should put the word curly in the index, the word curly will get into the index and there will be variations curly, curly, etc.
You can enable stemming for Russian, English or both languages
Morphology = stem_en
morphology = stem
morphology = stem_enru
You can also use the Soundex and Metaphone options, they allow you to use for English, taking into account the sound of words. I do not use these morphology algorithms in my work, so if someone knows a lot about them, I will be glad to read. For the Russian language, such algorithms would make it possible to obtain from the words “sun” and “sun” the normalized form “sun”, which is obtained based on the sound and pronunciation of these words.
Morphology = stem_enru, Soundex, Metaphone
You can also connect external engines for morphology or write your own.
wordforms
Allows you to connect your own dictionaries of word forms, is well used on specialized thematic sites, has a good example in the documentation.Core 2 duo > c2d
e6600 > c2d
core 2duo > c2d
Allows you to find an article about core 2 duo for any search query from model to name variations.
hemp > weed
dope > weed
my charm > weed
freedom grass > weed
what to smoke > weed
there is what > weed
And this dictionary will allow your user to easily find information about weed on the site.
For word forms, files in the ispell or MySpell format are used (which can be done in Open Office)
Wordforms = /usr/local/sphinx/data/wordforms.txt
enable_star
Allows you to use asterisks in queries, for example, the query *pr* will find prospectus, hello, approximation, etc.enable_star = 1
expand_keywords
Automatically expands the search query to three queriesRunning -> (running | *running* | =running)
Just a word with morphology, a word with asterisks and a complete word match. Previously, this option was not there, and in order to search with asterisks, I had to manually make an additional request, now everything is included with one option. Also, a full match will automatically be higher in the search results than a search with asterisks and morphology.
Expand_keywords = 1
index_exact_words
Allows, along with the morphologically normalized form, to store the original word in the index. This greatly increases the size of the index, but given the previous option, it allows you to return results more relevant.For example, there are three words “melon”, “melon”, “melon” without this option, all three words will be stored in the index as melons and the query “melon” will be issued in the order they were added to the index, that is, “melon”, “melon” , "melon".
If you enable the expand_keywords and index_exact_words options, then the query “melon” will have more relevant results for “melon”, “melon”, “melon”.
index_exact_words = 1
min_infix_len
Allows you to index parts of the word infixes, and search them using *, such as search*, *search and *search*.For example, if min_infix_len = 2 and the words “test” get into the index, they will be stored in the index “te”, “eu”, “st”, “tes”, “eats”, “test” and upon request “eu” will be found this word.
I usually use
Min_infix_len = 3
A smaller value generates too much garbage, and remember that using this option greatly increases the index.
min_prefix_len
It is a child of min_infix_len and does pretty much the same thing except it keeps word beginnings or prefixes.For example, if min_infix_len = 2 and the words “test” get into the index, “te”, “tes”, “test” will be stored in the index and this word will be found by the request “eu”.
min_prefix_len = 3
min_word_len
Minimum word size for indexing, default is 1 and indexes all words.
I usually use
min_word_len = 3
Smaller words usually do not carry a semantic load.
html_strip
Strips all html tags and html comments. This option is relevant if you are building your google/yandex based on sphinxsearch. We launched the spider, parsed the site, drove it into the database, set the indexer and this option will allow you to get rid of junk in the form of html tags and search only for the content of the site.Unfortunately, I didn’t use it myself, but the documentation says that it can mess with all sorts of xml and non-standard html (for example, anywhere with opening and closing tags, etc.).
I will be glad to any questions and clarifications.
offsite
Search options are common in web pages and we can give the option to visitors to search for keywords to locate the article or data within a site. Here we will apply the search to an existing table. The visitors will enter one or more than one keywords in the search box and we have to develop a query based on the selection of the visitor. Here we will discuss how to create a sql query to apply to the database with multiple keywords. Basically we will focus on construction of sql using keywords and will not go into details of getting the result or displaying in proper format with page breaks . Here we will give the option to the visitor to search for exact match or any where match on the table. For this we will use our student table and apply the search to the name field of the MySQL table. Based on the selected type of search we will generate the sql. Here are the steps required to develop a keyword search. Once the form is submitted check if the visitor has asked for exact match or any where match using if condition. Before that we will remove the blank space from left and right of the search sting by using ltrim and rtrim functions . $search_text=ltrim($search_text); $search_text=rtrim($search_text); If the visitor has asked for exact match then create the query using simple . Else .. If the visitor has asked for anywhere matching of keywords then read the search term and break it into array of keywords using explode command . Then loop through all the element of the array of words and create the sql command using like command for each word or the element of the array. Here is the code for this. $kt=explode(" ",$search_text);//Breaking the string to array of words // Now let us generate the sql while(list($key,$val)=each($kt))( if($ val<>" " and strlen($val) > 0)($q .= " name like "%$val%" or ";) )// end of while You can see we have broken the search text using explode command and then looped through the keywords. Here using one if condition we have taken care that blank space are removed in formatting the sql string. This way we will be adding one sql like command with OR combination for each word used. We will be adding each like command to the string with an SQL OR command. This way we will end with an extra OR command. This extra OR command we can remove from the end by using substr and strlen string functions. $q=substr($q,0,(strlen($q)-3)); In the above line we have first calculated the length of the string by using strlen and then used that value inside the substr function after subtracting 3 from it. The 3 is subtracted as length of OR with one blank space is 3. This way we will get the string after removing 3 chars from the end. (that is extra OR with a blank space) $sql=preg_replace("/AND/","WHERE",$sql,1); Above code will replace first occurance of AND with WHERE We can remove extra char from the end of the string by using PHP rtrim function Once this sql is formatted then we can print it to the screen to check the syntax of the sql to match our requirement. Beyond this point you can use any server side script to collect the results from the table. You can read the tutorial on how to display data from mysql using php . You can download the PHP version of this tutorial in a zip file at the end of this page. Here is the code till now.0)( if(!ctype_alnum($search_text))( echo "Data Error"; exit; ) ) ////////// Displaying the search box ///// echo "
| "; echo " "; echo " | |||||||||||||||||||||
| "; if(isset($todo) and $todo=="search")( $type=$_POST["type"]; $search_text=ltrim($search_text); $search_text=rtrim($search_text); if( $type<>"any")( $query="select * from student where name="$search_text""; )else( $kt=explode(" ",$search_text);//Breaking the string to array of words // Now let us generate the sql while(list($key,$val)=each($kt))( if($val<>" " and strlen($val) > 0)($q .= " name like "%$val%" or ";) )// end of while $q=substr($q,0,(strLen($q )-3)); // this will remove the last or from the string. $query="select * from student where $q"; ) // end of if else based on type value echo " $query "; $count=$dbo->prepare($query); $count->execute(); $no=$count->rowCount(); if($no > 0)(echo " No of records = ". $no." "; echo "
| "; echo " Full records here "; $query="select * from student"; echo "
| ||||||||||||||||||||
By Ibrahim Diallo
Published Jul 2 2014 ~ 16 minutes readSearch is an important feature on a website. When my few readers want to look for a particular passage on my blog, they use the search box. It used to be powered by Google Search, but I have since then changed it to my own home-brewed version not because I can do better but because it was an interesting challenge.
If you are in a hurry and just want your site to be searchable, well do what I did before, use Google.
// In search.php file $term = isset($_GET["query"])?$_GET["query"]: ""; $term = urlencode($term); $website = urlencode("www.yourwebsite.com"); $redirect = "https://www.google.com/search?q=site%3A($website)+($term)"; header("Location: $redirect"); exit;
What it does is pretty simple. Get the term passed by the user, and forward it to Google search page. Limit the search result to our current domain using the site: keyword in the search query. All your pages that are indexed by Google will be available through search now. If you do want to handle your search in house however, then keep reading.
Homemade Search Solution
Before we go any further, try using the search box on this blog. It uses the same process that I will describe below. If you feel that this is what you want then please continue reading.
This solution is catered to small websites. I make use of LIKE with wild cards on both ends, meaning your search cannot be indexed. This means the solution will work fine for your blog or personal website that doesn't contain tons of data. Port it to a bigger website and it might become very slow. MySQL offers Full Text Search which is not what we are doing here.
Note: If you have 5000 blog posts you are still fine. .
We will take the structure of this blog as a reference. Each blog post has:
- A title p_title
- A url p_url
- A summary p_summary
- A post content p_content
- And categories category.tagname
For every field that matches with our search term, we will give it a score. The score will be based on the importance of the match:
// the exact term matches is found in the title $scoreFullTitle = 6; // match the title in part $scoreTitleKeyword = 5; // the exact term matches is found in the summary $scoreFullSummary = 5; // match the summary in part $scoreSummaryKeyword = 4; // the exact term matches is found in the content $scoreFullDocument = 4; // match the document in part $scoreDocumentKeyword = 3; // matches a category $scoreCategoryKeyword = 2; // matches the url $scoreUrlKeyword = 1;
Before we get started, there are a few words that do not contribute much to a search that should be removed. Example "in","it","a","the","of" ... . We will filter those out and feel free to add any word you think is irrelevant. Another thing is, we want to limit the length of our query. We don't want a user to write a novel in the search field and crash our MySQL server.
// Remove unnecessary words from the search term and return them as an array function filterSearchKeys($query)( $query = trim(preg_replace("/(\s+)+/", " ", $query)); $words = array(); // expand this list with your words. $list = array("in","it","a","the","of","or","I","you", "he","me","us","they","she","to","but","that","this","those","then"); $c = 0; foreach(explode(" ", $query) as $key)( if (in_array($key, $list))( continue; ) $words = $key; if ($c >= 15)( break; ) $c++ ; ) return $words; ) // limit words number of characters function limitChars($query, $limit = 200)( return substr($query, 0,$limit); )
Our helper functions can now limit character count and filter useless words. The way we will implement our algorithm is by giving a score every time we find a match. We will match words using the if statement and accumulate points as we match more words. At the end we can use that score to sort our results
Note: I will not be showing how to connect to MySQL database. If you are having problems to efficiently connect to the database I recommend reading this .
Let "s give our function a structure first. Note I left placeholders so we can implement sections separately.
Function search($query)( $query = trim($query); if (mb_strlen($query)===0)( // no need for empty search right? return false; ) $query = limitChars($query) ; // Weighing scores $scoreFullTitle = 6; $scoreTitleKeyword = 5; $scoreFullSummary = 5; $scoreSummaryKeyword = 4; $scoreFullDocument = 4; $scoreDocumentKeyword = 3; $scoreCategoryKeyword = 2; $scoreUrlKeyword = 1; $keywords = filterSearchKeys( $query); $escQuery = DB::escape($query); // see note above to get db object $titleSQL = array(); $sumSQL = array(); $docSQL = array(); $categorySQL = array (); $urlSQL = array(); /** Matching full occurrences PLACE HOLDER **/ /** Matching Keywords PLACE HOLDER **/ $sql = "SELECT p.p_id,p.p_title,p.p_date_published,p. p_url, p.p_summary,p.p_content,p.thumbnail, ((-- Title score ".implode(" + ", $titleSQL).")+ (-- Summary ".implode(" + ", $sumSQL) .")+ (-- document ".implode(" + ", $docSQL).")+ (-- tag/category ".implode(" + ", $categorySQL).")+ (-- url ". implode(" + ", $urlSQL).")) as relevance FROM post p WHERE p.status = "published" HAVING relevance >
In the query, all scores will be summed up as the relevance variable and we can use it to sort the results.
Matching full occurrences
We make sure we have some keywords first then add our query.
If (count($keywords) > 1)( $titleSQL = "if (p_title LIKE "%".$escQuery."%",($scoreFullTitle),0)"; $sumSQL = "if (p_summary LIKE "%" .$escQuery."%",($scoreFullSummary),0)"; $docSQL = "if (p_content LIKE "%".$escQuery."%",($scoreFullDocument),0)"; )
Those are the matches with higher score. If the search term matches an article that contains these, they will have higher chances of appearing on top.
Matching keyword occurrences
We loop through all keywords and check if they match any of the fields. For the category match, I used a sub-query since a post can have multiple categories.
Foreach($keywords as $key)( $titleSQL = "if (p_title LIKE "%".DB::escape($key)."%",($scoreTitleKeyword),0)"; $sumSQL = "if (p_summary LIKE "%".DB::escape($key)."%",($scoreSummaryKeyword),0)"; $docSQL = "if (p_content LIKE "%".DB::escape($key)."% ",($scoreDocumentKeyword),0)"; $urlSQL = "if (p_url LIKE "%".DB::escape($key)."%",($scoreUrlKeyword),0)"; $categorySQL = "if ((SELECT count(category.tag_id) FROM category JOIN post_category ON post_category.tag_id = category.tag_id WHERE post_category.post_id = p.post_id AND category.name = "".DB::escape($key)."") > 0,($scoreCategoryKeyword),0)"; )
Also as pointed by a commenter below, we have to make sure that the these variables are not empty arrays or the query will fail.
// Just incase it"s empty, add 0 if (empty($titleSQL))( $titleSQL = 0; ) if (empty($sumSQL))( $sumSQL = 0; ) if (empty($docSQL))( $docSQL = 0; ) if (empty($urlSQL))( $urlSQL = 0; ) if (empty($tagSQL))( $tagSQL = 0; )
At the end the queries are all concatenated and added together to determine the relevance of the post to the search term.
// Remove unnecessary words from the search term and return them as an array function filterSearchKeys($query)( $query = trim(preg_replace("/(\s+)+/", " ", $query)); $words = array(); // expand this list with your words. $list = array("in","it","a","the","of","or","I","you", "he","me","us","they","she","to","but","that","this","those","then"); $c = 0; foreach(explode(" ", $query) as $key)( if (in_array($key, $list))( continue; ) $words = $key; if ($c >= 15)( break; ) $c++ ; ) return $words; ) // limit words number of characters function limitChars($query, $limit = 200)( return substr($query, 0,$limit); ) function search($query)( $query = trim ($query); if (mb_strlen($query)===0)( // no need for empty search right? return false; ) $query = limitChars($query); // Weighing scores $scoreFullTitle = 6; $ scoreTitleKeyword = 5; $scoreFullSummary = 5; $scoreSummaryKeyword = 4; $scoreFullDocument = 4; $scoreDocumentKeyword = 3; $scoreCategoryKeyword = 2; $scoreUrlKeyword = 1; $keywords = filterSearchKeys($query); $escQuery = DB::escape($query); // see note above to get db object $titleSQL = array(); $sumSQL = array(); $docSQL = array(); $categorySQL = array(); $urlSQL = array(); /** Matching full occurences **/ if (count($keywords) > 1)( $titleSQL = "if (p_title LIKE "%".$escQuery."%",($scoreFullTitle),0)"; $sumSQL = "if (p_summary LIKE "%".$escQuery."%",($scoreFullSummary),0)"; $docSQL = "if (p_content LIKE "%".$escQuery."%",($scoreFullDocument), 0)"; ) /** Matching Keywords **/ foreach($keywords as $key)( $titleSQL = "if (p_title LIKE "%".DB::escape($key)."%",($scoreTitleKeyword ),0)"; $sumSQL = "if (p_summary LIKE "%".DB::escape($key)."%",($scoreSummaryKeyword),0)"; $docSQL = "if (p_content LIKE "% ".DB::escape($key)."%",($scoreDocumentKeyword),0)"; $urlSQL = "if (p_url LIKE "%".DB::escape($key)."%",( $scoreUrlKeyword),0)"; $categorySQL = "if ((SELECT count(category.tag_id) FROM category JOIN post_category ON post_category.tag_id = category.tag_id WHERE post_category.post_id = p.post_id AND category.name = "". DB::escape($key)."") > 0,($scoreCategoryKeyword),0)"; ) // Just incase it"s empty, add 0 if (empty($titleSQL))( $titleSQL = 0; ) if (empty($sumSQL))( $sumSQL = 0; ) if (empty($docSQL))( $docSQL = 0; ) if (empty($urlSQL))( $urlSQL = 0; ) if (empty($tagSQL))( $tagSQL = 0; ) $sql = " SELECT p.p_id,p.p_title,p.p_date_published,p.p_url, p.p_summary,p.p_content,p.thumbnail, ((-- Title score ".implode(" + ", $titleSQL). ")+ (-- Summary ".implode(" + ", $sumSQL).")+ (-- document ".implode(" + ", $docSQL).")+ (-- tag/category ".implode (" + ", $categorySQL).")+ (-- url ".implode(" + ", $urlSQL).")) as relevance FROM post p WHERE p.status = "published" HAVING relevance > 0 ORDER BY relevance DESC,p.page_views DESC LIMIT 25"; $results = DB::query($sql); if (!$results)( return false; ) return $results; )
Now your search.php file can look like this:
$term = isset($_GET["query"])?$_GET["query"]: ""; $search_results = search($term); if (!$search_results) ( echo "No results"; exit; ) // Print page with results here.
We created a simple search algorithm that can handle a fair amount of content. I arbitrarily chose the score for each match, feel free to tweak it to something that works best for you. And there is always room for improvement.
It is a good idea to track the search term coming from your users, this way you can see if most users search for the same thing. If there is a pattern, then you can save them a trip and just cache the results using memcached.
If you want to see this search algorithm in action, go ahead and try looking for an article on the search box on top of the page. I have added extra features like returning the part where the match was found in the text. Feel free to add features to yours.
Did you like this article? You can subscribe to read more awesome ones. .
On a related note, here are some interesting articles.
It is time to deal with mysql_* functions once and for all. These methods are deprecated and slow. The time to upgrade has long passed yet we still see it everywhere. Since I cannot force every author to update their tutorial and blogs, I decided to write a post to hopefully rank better and provide the essential information to help new comers.
Making your own website shouldn't be too difficult. Hosting companies like Godaddy or Hostgator make it super easy for anyone to get started; they allow you to create a whole website without ever writing code. For most people, it is plenty to run a WordPress blog. If this is what you are looking for you should head to Godaddy.com right now. We are done here. But on the other hand, if you want to have control and not be limited by the short comings of a shared hosting without busting your wallet, you have come to the right place.
Vim is my favorite text editor on the terminal. After playing for a little while with nano and emacs , I finally settled with vim for its simplicity (bare with me please). Although it can be customized and used like an entire IDE, I use it mostly for editing files on my servers and making small but crucial changes. Let's not get into Editor war and get started.
Comments(45)
Zaryel Aug 12 2015:
Ian Mustafa Sep 26 2015:
Rob Sep 29 2015:
adeem Feb 11 2016:
Ivan Venediktov Apr 9 2016.
When automatic parsers work, there is a natural need to match the text with its keywords. This is necessary for the search engine optimization of the article.
For example, we parsed an ad from a board, it sells an Apple Iphone 4s, used. Small description text with features and price.
From this text it is necessary to highlight the key words. Filling them with automatic parsing will be expensive. Therefore, you need to use scripts to select keywords.
First, you need to make a list of all the words used in the text, sorted by the frequency of their use. Throwing away short words (less than 5 characters) and various auxiliary parts of speech, such as pronouns and adverbs.
Once upon a time, I used Danneo CMS to develop websites. So, it had a great function of highlighting keywords in the text. Just for this, I downloaded a new version of Danneo, by the way, it was updated to version 0.5.5
Here is the function
Function seokeywords($contents,$symbol=5,$words=35)( $contents = @preg_replace(array(""<[\/\!]*?[^<>]*?>"si",""([\r\n])[\s]+"si",""&(1,6);"si",""(+)"si"), array("","\\1 "," "," "),strip_tags($contents)); $rearray = array("~","!","@","#","$","%","^","&","*","(",")", "_","+", "`",""","№",";",":","?","-","=","|","\""," \\","/", "[","]","(",")",""",",",".","<",">","\r\n","\n","\t",""","""); $adjectivearray = array("th","th","th","th"," oy","th","oh","mi","th","her","th","them","th","how","for","what","or" , "this", "these", "all", "you", "they", "it", "still", "when", "where", "this", "only", "already"," you","no","if","necessary","everything","so","his","than", "at","even","me","is","only" ,"very", "now","exactly","usually"); $contents = @str_replace($rearray," ",$contents); $keywordcache = @explode(" ",$contents); $rearray = array(); foreach($keywordcache as $word)( if(strlen($word)>=$symbol && !is_numeric($word))( $adjective = substr($word,-2); if(!in_array( $adjective,$adjectivearray) && !in_array($word,$adjectivearray))( $rearray[$word] = (array_key_exists($word,$rearray)) ? ($rearray[$word] + 1) : 1; ) ) ) @arsort($rearray); $keywordcache = @array_slice($rearray,0,$words); $keywords = ""; foreach($keywordcache as $word=>$count)( $keywords.= "," .$word; ) return substr($keywords,1); )
from the above typed text, the function produces the following set
Array,rearray,contents,keywordcache,words,keywords,words,adjectivearray,symbol,replace,words,this,here,
Danneo,text,function,need,key,foreach,text,key,substr,need,adjective,displays,
was, gorgeous, selection, downloaded, for the sake of, Only, her, typed, set, long ago
not bad, is it?!
Further, you can do the following: sift all these words through the database of pre-prepared keywords. And leave only the ones you need. I have been creating such a database for several years manually. This is the database of tags that underpin each article.
So by keywords, you can easily restore information about the article. Those. The search engine will have a concrete idea of what your page contains. Similar to
Keywords (they are also called search queries, or keys) are needed in order to optimize existing pages of the site for them, use queries when creating new materials on your site, and also to compose the texts of external links and articles when placing them on others. resources.
There are paid and free methods for word selection. Today I will talk about paid solutions that include online services, browser extensions, as well as desktop programs (free and paid with a free trial period).
Free online services for collecting search queries
Perhaps this is the best service for selecting keywords for promotion on Google, because it is a “native” Google service. It is suitable for both Runet and the English-speaking Internet, you just need to select the desired country and language:
A lot of services and programs for selecting words use the data of this particular service.
This service has options that allow you to select queries for both desktops and laptops, as well as for mobile devices:
I advise you to activate all columns for information on keywords. To do this, on the right side of the page, click on the “Columns” button (Columns) and check all the checkboxes: Level of competition (Competition), the number of requests per month for the whole world (Global Monthly Searches) and target regions (Global Local Searches), the share of ad impressions (Ad Share), Google Search Network, First Page Impressions (Search Share), Approximate CPC (cost-per-click bid), Local Search Trends graph), Web page content (Extracted From Webpage).
Here is a screenshot in the English interface that I prefer to work in:
The columns in the displayed menu can be dragged and swapped with the mouse, which is very convenient.
I also recommend playing with the match type in the left column to get the desired results and additional ideas for search queries (try collecting keys by ticking “Exact” (Exact), “Wide” (Broad) and “Phrase” in turn).
The received data can be saved in CSV format (as well as in several other formats), to do this, click on the “Save to file” button. I sincerely recommend using this service as it has all the features you need to collect keywords.
A useful and free keyword collection service that uses Google Suggest and other resources. You can select the country and source (Web, News, Shopping, Video or Recipes). Ubersuggest allows you to receive search queries from both organic and vertical searches. This resource works like this: it takes the request you enter, adds a letter or number to it, and collects all possible requests. Thus, you get the opportunity to find more ideas for keywords, especially for low-frequency queries.
For example, I collected 344 search queries for the phrase “search engine optimization” and 377 for “photoshop tutorials”.
Saving the received data is done in an original way. Click on the “Select all keywords” button. By the way, how do you like my new watermark? As they say in Florida - Style! :
Then click on the “Get” button on the right side:
And a window will appear with all the collected keywords.
There are two options for using the Youtube keyword search service: enter the desired query or insert a link to the video for which you want to select words and phrases.
I found that the Youtube Keyword Tool service does not work correctly and for many requests it shows a message that there is not enough data. I came up with this solution - register or log into your Google AdWords account and paste a link to a video that is relevant to your keywords. Then set the price to $0.01 per view and you will get all possible search phrases. I also advise you to play with the Broad, Phrase and Exact match type (Wide, Phrase and Exact) for maximum results.
Another useful keyword collection tool from Google. It allows you to compare data across categories, countries, and time periods. You can apply Web Search, Image Search, News and Product Search filters.
Google Insights has another interesting feature - you can see the dynamics of requests over time and on the world map (click on the “View change over time” item, you can even turn on animation there):
For example, I found out that the query “search engine optimization” in 2004 was most popular in the US, and now the country where this phrase is most requested is India.
A cool service for collecting search phrases, which is also free. It shows data from the world's popular Internet resources (Google, Amazon, Wikipedia, Answers.com, Yahoo, Bing, Youtube, Netflix, Ebay, Buy.com, Weather.com) and displays phrases in an original manner as you type your request in it. Try it, you will like it!
I found a tiny button to save the results, it is in the upper left side.
The well-known analytical resource Alexa.com, with which you can find search queries for a particular site for free. We go to Alexa.com, select the Site Info tab and enter the address of the site we are interested in. Then click on the Get Details button:
On the next page, select the Search Analytics tab:
At the bottom of the page will be a list of the most popular search queries for which this site receives visitors from search engines.
The Bing search query selection service allows you to collect data from organic search and study statistics for the last 6 months for free. You can filter the received requests by language and country. In order to use this service, you need to register with it.
Another free keyword collection service from Google. You can compare several queries, for this write them, separating them with a comma. You can compare up to 5 queries.
An excellent search query generator that takes data from Google, Yahoo and Wordtracker. For example, I got 46 key phrases for “photoshop tutorials”.
The name of this service can be translated as "Yahoo Guesses". All keyword data is obtained from the Yahoo search engine. You can find specific searches for sports leagues, sports teams, and athletes. There is a function for comparing requests that men and women type. You can also filter keys by country.
And another free tool from Google. This free online tool finds search patterns which correspond with real-world trends, which is very useful for SEO specialists and website owners.
Google Correllate also shows search activity maps. For example, I use it to find out the popularity of a particular query in various US states. Just hover over the desired state, and you will see information on it. I pointed to my beloved Florida, where I lived for 4 years:
A popular free tool among foreign optimizers. The data is taken from the Wordtracker service. In order to start using it, you need to create an account. You can save received requests in CSV format.
I met a couple of times that in Runet the name of this tool is translated as “Stupid generator”. It's actually a typo generator. Some of the queries users have always entered, are entering and will enter with typos. Therefore, typos can be used to get additional targeted visitors. This tool just allows you to get key queries with typos.
Browser extensions for searching keywords
Search queries can be received not only from various online services. You can also study those pages of competitors' sites that already occupy the first places on topics of interest to you and get keys from them based on the density of requests. This method allows you to get additional ideas about what words and phrases can still be used in the text of the pages you promote. There are browser extensions for this, which I will discuss next.
I have been actively using SeoQuake since 2007. This extension is available for Google Chrome, Mozilla Firefox, Opera and Safari browsers.
With the SeoQuake extension, it is very easy to analyze keyword density on any page of the site. To do this, go to the page you are interested in, click on the SeoQuake icon and select Keyword Density:
On the next page, you will see detailed information about which keywords are being used to optimize this page:
You can analyze 1-word, 2-word, 3-word, and 4-word queries (very useful for low-frequency searches), as well as a keyword cloud:
I found a video that shows this process in detail in SeoQuake:
KGen (short for Keyword Generator) is an SEO addon for the Mozilla Firefox browser that allows you to find out which search queries are used on a particular page to optimize it. The essence is the same as the previous extension.
KGen crawls the page and displays information about how many times a particular request occurs, as well as its weight and the average position of the request on the page.
To select keywords, simply go to the desired page, click on the KGen icon (I have it in the lower right part of the Firefox browser) and click the “Scan” button in the window that appears.
Free versions of paid online services for collecting search queries
This is one of my favorite tools for researching sites and selecting keywords when promoting on Google (I use the Professional plan). It has a fairly functional free version, with which you can get very valuable information about the queries and links of competitors.
For example, you can find out what queries the competitor you are interested in promotes this or that internal page of their site. To do this, go to the SERP's Analysis tab, select Organic Keywords and enter the address of the desired page. You will receive 10 search queries for which this page is in Google issuance for free. Data is available not only for Google, but also for Bing and Yahoo.
I did a detailed 40+ minute video review of Ahrefs.com features in this post: How to Explore Competitor Sites with Ahrefs.com - Video Tutorial.
The free version of Wordtracker generates up to 100 search queries. Please note that it shows the frequency of impressions per day. If you need data for a month (like me), then just multiply the results by 30.
Another useful tool from the Wordtracker service. Enter a query and get 100 questions that match it. This is a good help when looking for new ideas for keywords and writing articles. Questions can be used in the preparation of assignments for copywriters.
Service from a well-known blogger in Runet Dimka. Actual Keywords allows you to get 100 keywords for free for any query. Also available for free download are selections of keywords on several topics (Auto Loan, Digital Camera, What Is and How To).
The popular service Semrush allows you to enter 10 queries per day for free and get 10 results for each query. This service has its own database, which consists of 95 million keywords and 45 million sites.
Like other free versions of paid services, Keyword Discovery returns 100 search queries. The data is collected from over 200 resources around the world.
Another way to get 100 requests for free. Data is available from Google for several countries: AU, BR, CA, DE, ES, FR, IT, RU, UK and US.
I found an interesting feature in Keyword Eye - a cloud of search queries. Here's what it looks like:
Wordpot shows data from various search engines by frequency of impressions per day.
For free, this service allows you to create 1 project and make 25 requests per day. Synonyms, related queries and associated words are also limited to 5 per day.
If you participate in the Amazon.com affiliate program, then this service will allow you to find search queries from Amazon, which is especially valuable for finding specific low-frequency queries. Please note that in the free version you can only enter 3 requests per day.
Like the other services listed above, WordStream gives us 100 requests for free. Disappointing that the received data cannot be downloaded. But you can send them to your e-mail. Although I personally don’t like this moment - what if a newsletter that I don’t need comes?
This service collects data from eBay.com. Enter the required query and get such a table (there is still information there, I did not capture it in the screenshot in order to show this particular table in detail):
This service reminds me of a closed project from Google - Wonder Wheel. Looks like it, doesn't it?
Terapeak allows you to find search queries for specific countries. There is a free trial period for 7 days.
And one more way to get 100 keywords for free. I did not find a way to save the received data, because this service is inferior to those that I described above.
Niche Bot issues 20 search queries per day.
And this one is only 10. The fewer requests the service issues for free, the less I write about it. Will issue 1 request - just put a dot instead of a description.
And this tool issues 100 requests. For example, I entered “Photoshop tutorials” and got 110 phrases. But after entering the second request, I received a message about reaching the daily limit “You have reached your daily limit”. In the free version, you can't download the collected keywords report, but you can select it with the mouse and copy it, although this is not so convenient.
This service allows you to find out the Keyword Effectiveness Index (KEI), that is, the index of the effectiveness of a keyword, as well as collect queries from sites you are interested in. It also has a module for collecting keywords for which a particular site is in the search results. Wordze has a free trial of 30 days.
SpyFu allows you to collect queries that promote any site you are interested in. It's a popular competitor research tool on the English-speaking Internet, but it provides very little information in the free version.
Free Keyword Search Software
Useful free program for Windows. The data is collected from the Google word search service.
This program allows you to select search queries directly in the Microsoft Excel interface, which is very convenient for many users. The data is collected from Bing and Yahoo search engines and includes relevance, cost history, impression frequency, and country data.
To be honest, when I installed the Microsoft Advertising Intelligence program, I did not immediately understand how to start it. There is no shortcut on the desktop, I did not find it in the list of programs either. And it was only after I launched Excel that I saw a new tab. I took a screenshot above where you can see it.
Paid Word Search Software with Free Trial
The Good Keywords project has created another solution for searching for keywords - the Keyword Strategy Studio program. It allows you to find new ideas for search queries that are based on search data, trends and popularity. Keyword Strategy Studio has a fully functional trial version that you can use for 30 days.
Market Samurai is a very popular program among foreign SEOs. It is paid, but there is a free trial period. The most remarkable thing is that the keyword selection module in Market Samurai continues to work normally even after the end of the trial period, that is, over the network, this function in the program can be used for free for an unlimited amount of time.
Keyword Researcher is literally "Keyword Researcher". This program is well suited for finding low-frequency search queries with low competition and works on Windows and Apple OS X. The screenshot shows that you can set a search query template by replacing one word or phrase with an asterisk, and the program will select all possible keywords. For example, we make the following request:
How to * a camera
And the program selects 119 phrases according to this template. There is a free period. I suggest watching a video about its capabilities:
This program has a free version that has many interesting features. For example, I liked the Amazon keyword scraper module (for collecting keywords from Amazon.com). See screenshot:
You can set various filters when searching for keywords and phrases, you can remove brand names with one click, set the range of local query impressions and global impressions, as well as set parameters for the average cost per click (CPC). Just like in Google Adwords, you can specify the type of match (exact, broad, and phrase).
The Advanced Web Ranking program provides all its functionality during the entire free trial period of 30 days and works on Windows, Mac OS X and Linux operating systems.
The keyword selection module in Rank Tracker allows you to determine the level of competition, find misspelled words, etc. The data is collected from Google AdWords, Yandex Wordstat, Keyword Discovery, Wordtracker, Bing and Semrush. You can use the Rank tracker for free, but you will not be able to save projects and reports. The program works on Windows, Mac OS X and Linux.
I also want to mention two popular search query tools - Micro Niche Finder (program) and Keyword Ninja (PHP script), but they do not have a free trial period, so I did not include them in the list, I bring them for information. Micro Niche Finder has a period of 30 days, but you have to pay for it, so it's not free.
