.. cloudsearch_tut: =============================================== An Introduction to boto's Cloudsearch interface =============================================== This tutorial focuses on the boto interface to AWS' Cloudsearch_. This tutorial assumes that you have boto already downloaded and installed. .. _Cloudsearch: http://aws.amazon.com/cloudsearch/ Creating a Domain ----------------- >>> import boto >>> our_ip = '192.168.1.0' >>> conn = boto.connect_cloudsearch() >>> domain = conn.create_domain('demo') >>> # Allow our IP address to access the document and search services >>> policy = domain.get_access_policies() >>> policy.allow_search_ip(our_ip) >>> policy.allow_doc_ip(our_ip) >>> # Create an 'text' index field called 'username' >>> uname_field = domain.create_index_field('username', 'text') >>> # But it would be neat to drill down into different countries >>> loc_field = domain.create_index_field('location', 'text', facet=True) >>> # Epoch time of when the user last did something >>> time_field = domain.create_index_field('last_activity', 'uint', default=0) >>> follower_field = domain.create_index_field('follower_count', 'uint', default=0) >>> domain.create_rank_expression('recently_active', 'last_activity') # We'll want to be able to just show the most recently active users >>> domain.create_rank_expression('activish', 'text_relevance + ((follower_count/(time() - last_activity))*1000)') # Let's get trickier and combine text relevance with a really dynamic expression Viewing and Adjusting Stemming for a Domain -------------------------------------------- A stemming dictionary maps related words to a common stem. A stem is typically the root or base word from which variants are derived. For example, run is the stem of running and ran. During indexing, Amazon CloudSearch uses the stemming dictionary when it performs text-processing on text fields. At search time, the stemming dictionary is used to perform text-processing on the search request. This enables matching on variants of a word. For example, if you map the term running to the stem run and then search for running, the request matches documents that contain run as well as running. To get the current stemming dictionary defined for a domain, use the ``get_stemming`` method of the Domain object. >>> stems = domain.get_stemming() >>> stems {u'stems': {}} >>> This returns a dictionary object that can be manipulated directly to add additional stems for your search domain by adding pairs of term:stem to the stems dictionary. >>> stems['stems']['running'] = 'run' >>> stems['stems']['ran'] = 'run' >>> stems {u'stems': {u'ran': u'run', u'running': u'run'}} >>> This has changed the value locally. To update the information in Amazon CloudSearch, you need to save the data. >>> stems.save() You can also access certain CloudSearch-specific attributes related to the stemming dictionary defined for your domain. >>> stems.status u'RequiresIndexDocuments' >>> stems.creation_date u'2012-05-01T12:12:32Z' >>> stems.update_date u'2012-05-01T12:12:32Z' >>> stems.update_version 19 >>> The status indicates that, because you have changed the stems associated with the domain, you will need to re-index the documents in the domain before the new stems are used. Viewing and Adjusting Stopwords for a Domain -------------------------------------------- Stopwords are words that should typically be ignored both during indexing and at search time because they are either insignificant or so common that including them would result in a massive number of matches. To view the stopwords currently defined for your domain, use the ``get_stopwords`` method of the Domain object. >>> stopwords = domain.get_stopwords() >>> stopwords {u'stopwords': [u'a', u'an', u'and', u'are', u'as', u'at', u'be', u'but', u'by', u'for', u'in', u'is', u'it', u'of', u'on', u'or', u'the', u'to', u'was']} >>> You can add additional stopwords by simply appending the values to the list. >>> stopwords['stopwords'].append('foo') >>> stopwords['stopwords'].append('bar') >>> stopwords Similarly, you could remove currently defined stopwords from the list. To save the changes, use the ``save`` method. >>> stopwords.save() The stopwords object has similar attributes defined above for stemming that provide additional information about the stopwords in your domain. Viewing and Adjusting Stopwords for a Domain -------------------------------------------- You can configure synonyms for terms that appear in the data you are searching. That way, if a user searches for the synonym rather than the indexed term, the results will include documents that contain the indexed term. If you want two terms to match the same documents, you must define them as synonyms of each other. For example: cat, feline feline, cat To view the synonyms currently defined for your domain, use the ``get_synonyms`` method of the Domain object. >>> synonyms = domain.get_synsonyms() >>> synonyms {u'synonyms': {}} >>> You can define new synonyms by adding new term:synonyms entries to the synonyms dictionary object. >>> synonyms['synonyms']['cat'] = ['feline', 'kitten'] >>> synonyms['synonyms']['dog'] = ['canine', 'puppy'] To save the changes, use the ``save`` method. >>> synonyms.save() The synonyms object has similar attributes defined above for stemming that provide additional information about the stopwords in your domain. Adding Documents to the Index ----------------------------- Now, we can add some documents to our new search domain. >>> doc_service = domain.get_document_service() >>> # Presumably get some users from your db of choice. >>> users = [ { 'id': 1, 'username': 'dan', 'last_activity': 1334252740, 'follower_count': 20, 'location': 'USA' }, { 'id': 2, 'username': 'dankosaur', 'last_activity': 1334252904, 'follower_count': 1, 'location': 'UK' }, { 'id': 3, 'username': 'danielle', 'last_activity': 1334252969, 'follower_count': 100, 'location': 'DE' }, { 'id': 4, 'username': 'daniella', 'last_activity': 1334253279, 'follower_count': 7, 'location': 'USA' } ] >>> for user in users: >>> doc_service.add(user['id'], user['last_activity'], user) >>> result = doc_service.commit() # Actually post the SDF to the document service The result is an instance of `cloudsearch.CommitResponse` which will makes the plain dictionary response a nice object (ie result.adds, result.deletes) and raise an exception for us if all of our documents weren't actually committed. Searching Documents ------------------- Now, let's try performing a search. >>> # Get an instance of cloudsearch.SearchServiceConnection >>> search_service = domain.get_search_service() >>> # Horray wildcard search >>> query = "username:'dan*'" >>> results = search_service.search(bq=query, rank=['-recently_active'], start=0, size=10) >>> # Results will give us back a nice cloudsearch.SearchResults object that looks as >>> # close as possible to pysolr.Results >>> print "Got %s results back." % results.hits >>> print "User ids are:" >>> for result in results: >>> print result['id'] Deleting Documents ------------------ >>> import time >>> from datetime import datetime >>> doc_service = domain.get_document_service() >>> # Again we'll cheat and use the current epoch time as our version number >>> doc_service.delete(4, int(time.mktime(datetime.utcnow().timetuple()))) >>> service.commit()