Searching Daily Kos
Quick Search Tips
Search: enter words you want to find in the target collection. By default only targets containing all the words will be returned. Capitalization is ignored. If you want to find several words in a precise order (a phrase) put double quotes around them as in "this is a phrase". Single quotes are not permitted, and are removed. Although "AND" is assumed by default, "OR" and "NOT" are supported, as follows.
Multiple query words are by default understood to be joined by "AND". Certain characters have to be "escaped" by placing a back slash in front of them; to find "/", use "\/", to find "=", use "\=". Complex queries can be constructed using parentheses and restricted queries. Restricted queries: limiting the parts of the document to be searched. The available options are described here. For both stories and comments (documents):
For stories only:
For comments only:
Find: In order to choose what kind of documents to find, use the pull down menu to select one of the following. Each option is identified as either a database search, meaning that the primary database is searched using Scoop directly, or index search, meaning that a specially constructed index of words contained in documents is searched using swish-e. There are three different indices, one for diaries (aka user diaries), one for stories (aka front page stories), and one for comments. User diaries that are elevated to the front page are found in both story and diary indices.
From: and To: pull down menus restrict the time interval of the search. The default interval is from one day ago to most recent. Most recent may be as much as 5 minutes out of date. Sort by: The order of presentation of results. Not all choices are relevant to all searches. Story and Diary Search sort choices.
Comment search sort choices shared with stories and diaries are "Relevance", "Title", and "Time", which are as described above. Those specific to comment are "Cid", "Nrec", and "Ntroll".
More example queries
For comments only Everything past the "#" is a comment, not part of the query!
Searching Daily Kos with election tags (etags)You can now search using election tags as follows. etag=CT-SEN Etags with embedded spaces should be quoted. etag="CA-LT. GOV" You can only use one etag per search. Good election tags have a two letter postal state abbreviation hyphen seperated from either a house district number, or a statewide office abbreviation (SEN, GOV, Lt.GOV, SOS, etc). There is one exception, etag="2006 ELECTION" also works. Good etags are ones we already know about by finding them in the tag list of an existing diary or by appearing in one of the election roundup diaries. The etag is used to build a more complicated query that looks for the tag in tags, title, or body text (stories/diaries) or in subject and comment text (comments). Certain meta dairies are excluded, including open threads, jotter's daily lists, and sidinny's election roundup, and the top comments diaries.
Software used.The search facility currently installed is based on the swish-e search engine: it is fast (compiled C code), open source, and flexible (you can feed it documents out of a database using your own custom preproccessor). It is intended for document collections numbering a million or less, which will do for Daily Kos with under half a million documents (excluding comments). In addition, swish-e offers many of features most desired, including search of phrases and search within configurable document areas (title, author, tags, URLs). What is indexed. All Daily Kos documents from the beginning to within 5 minutes of the present moment are indexed. Older documents are stored in static yearly indices while more recent documents are indexed and reindexed, frequently at first, then less frequently as they age, until they roll over into a growing, static, permenant yearly index. The right mix of number of index files and frequency of update to obtain the optimal combination of ease of maintenance and speed of search may require further experimentation. At present three different series of indices are maintained, corresponding to (in order of increasing total number of documents) front page stories, user diaries, and comments. For 2005 the number of stories, diaries, and comments was 5097, 88943, and 693224, respectively, containing 56993, 371578, and 2945916 distinct words. Design Considerations. The utility of search to a community of users is delivered not simply by the results of the text search, but most critically by the ordering of the results such that highest "value" hits are displayed first. (Most search researchers know that no one looks beyond about the first handful of results; almost no one looks beyond the first page.) The key to arranging for best to come first is to understand, have access to, and use the meta data associated with the text documents searched. Meta Data. At Daily Kos, we have a number of sorts of meta data that are relevant to documents. First among these is the time of posting. What is the latest? When did they say that? Who was first? That sort of thing. So the one thing search has to get right is time, in the sense of being up to date, that is including the latest diaries, as well as in the sense of permitting searches to be organized by reference to the present time. Making it up to date. Because recommendations are only open for 24 hours after a diary is published, it is important that search indices be refreshed sufficiently often during that interval to provide reasonably up to date information on total recommendations and comments. Currently, diaries up to 6 hours old are refreshed every 5 minutes, and diaries 6 to 24 hours old are refreshed every 15 minutes. More Meta Data. Other important meta data associated with stories and diaries includes comments and recommendations. The number of recommendations offers insight, imperfect, but useful, into the degree of approval a diary has received from the Daily Kos community. The number of comments indicates the size of the discussion engendered. The two numbers together give an interesting insight into a diaries history at Daily Kos. Some user diaries are promoted to the front page, and some diarists ("front pagers") are allowed to post directly to the front page. The latter are not open for recommendation. In order to harmonize listings from searches which include both front page stories and user diaries, a plausible value for number of recommendations (83) is assigned to front page stories, but in lists, this number is indicated by "*" to avoid placing too much emphasis on this made up number. The search query. Search queries can contain individual words, all of which must be found in a text document to constitute a hit. Queries can also contain phrases, indicated by double quotes around the phrase. All words are converted to lower case while searching, so letter case is of no consequence. In addition, queries can include boolean terms, such as "and", "or", and "not" which can be used to construct more complex and perhaps more informative results. By default "and" is assumed between individual words in a query. Finally, queries or parts of queries can be restricted to only search within parts of a document or document metadata. The parts available are shown above, along with examples. Certain characters have special meaning to the search engine (open and close paren, double quote, equals sign). Only a restricted list of characters are expected to occur in a word. For the indices here at daily kos, the word characters are as follows.
All other characters (non word, non special) are silently replaced by spaces while searching. For instance searching with "Jerome a Paris", "Jerome.a.Paris", or "Jerome,a,Paris" all give the same result. This is important in certain cases, for example searching for the url http://riverbendblogspot.com is accomplished with the query site=("riverbend.blogspot.com"). The reasonable looking but incorrect query site=riverbendblog.blogspot.com doesn't work as expected because in the absence of the double quotes and parentheses, replacement of "." with " " and the default boolean AND understood between search words makes it equivalent to site=riverbendblog AND blogspot AND com, which returns as hits only documents with riverbendblog in a url, and blogspot and com in the body text. More on Search
If you are intriqued by search and would
like to know more about the basics of how it is done, this is a
good introduction.
On Search. Basic basics. by Tim Bray. Since I've emphasized the importance of meta data, I'll also point to another article from the same series, On Search. SearchMeta, with some further thoughts on things meta. While I'm on the subject of underlying technologies, let me end this section by pointing to an article by Tim Bray on Ajax, the other new addition here at Daily Kos. In it, he also gives credit to Jesse James Garrett for the Ajax acronym in the process of thoroughly describing it.
Daily Kos is an amazing place, community, phenomenon. It has been my
privilege to have the opportunity to contribute.
-- jotter |
Recommended
- Primum non nocere (first, or above all, do no harm) is a Latin expression generally applied to the medical profession. However, I viewed it as equally applicable to the practice of law and often reminded myself of it in my own practice. Trump’s...278 comments 628 Recs
- There is an update on what is now officially the Worst Job in Russia — night watchman at an oil refinery. Salavat is in Bashkortostan and is north of Orenburg, where all the bad flooding took place. The drone that carried out the strike was a converted...25 comments 210 Recs
- But for want of 6 month’s worth of active and intense support from the United States this Russo-Ukrainian War would have a totally different color today. Ukrainian drones are now flying deep into Russia unmolested and tearing up valuable Russian...110 comments 444 Recs
- nep·o·tism | noun The practice among those with power or influence of favoring relatives or friends, especially by giving them jobs ------------------------------------------- Barron Trump, former President Donald Trump’s youngest child, who will...76 comments 93 Recs
- Heritage Foundation Mocks ‘Socialist’ Critics of Project 2025s Authoritarian Mandate For Leadership The Heritage Foundation, the country’s most powerful right wing think-tank, has come under intense scrutiny by journalists and progressive activists...85 comments 154 Recs
- Update: The bodycam footage shown by the Sheriff does show the police announced themselves. It also appears that Roger Fortson did open the door. Roger Fortson, who was Black, was killed in a home invasion. His home was invaded. By the police. Reports...197 comments 380 Recs
- I’ve been reeling over the fact moisture content in the atmosphere is accelerating. If you haven’t yet read it, you can find it here. Southern Brazil has been hit hard by record rainfall for weeks. There has been a pause as the rain bands moved and...181 comments 565 Recs
- WAPO. wapo.st/… So trump has promised to tank all industries that affect the profit margin of big oil. From article: Trump’s response stunned several of the executives in the room overlooking the ocean: You all are wealthy enough, he said, that you...149 comments 285 Recs
- It’s extremely early. I don’t believe that polls are meaningful predictors of the eventual outcome of a presidential election when they are this far out from November. Polls will only potentially be meaningful when we are past Labor Day, especially...88 comments 134 Recs
- Life Lessons and Other Bits from May Birthday Kids “Don't say things. What you are stands over you the while, and thunders so that I cannot hear what you say to the contrary.” —Ralph Waldo Emerson "When the Republican candidate inscribes the slogan...108 comments 197 Recs
- The Republican Party, in concert with their aspiring authoritarian Dear Leader, Donald Trump, has long been fiercely opposed to the basic tenets of democracy. They impose flagrantly gerrymandered redistricting, engage in overt voter suppression...25 comments 86 Recs
- In two weeks I’ll be heading off to Ireland for my first trip since COVID. Going with my daughter who has traveled frequently for business and vacations since 2020. She says she always masks up ...67 comments 20 Recs
Recent Diaries
- A veteran reporter speaks out; DHS marshals election defenses Lara Trump lies about ballots Johnson lies about 'illegals' voting; Also: MTG's spectacular fail Stormy wins the day Biden finally cuts off Israel... On today's BradCast: Why we're taking...1 comments 0 Recs
- I've been reading comments in Times of Israel strings attached to articles about Biden's decision to temporarily withhold 2,000 pound bombs, and other higher intensity offensive weaponry, to reduce the chance that Netanyahu will to do to Rafah what...3 comments 5 Recs
- “Pennsylvania Sen. John Fetterman, a staunch supporter of Israel, said that he disagreed with Biden’s plan to withhold some weapons shipments to Israel, warning the move “demonstrates to Hamas that they’re winning the PR war.” “ CNN quote After saying...31 comments 10 Recs
- Jack Schlossberg, grandson of President Kennedy, mercilessly mocked his cousin RFK Jr. AKA Wormboy in a series of Instagram videos. DK doesn’t embed Insta videos gracefully, so I am linking to the Boston.com story that has all the videos...2 comments 8 Recs
- In two weeks I’ll be heading off to Ireland for my first trip since COVID. Going with my daughter who has traveled frequently for business and vacations since 2020. She says she always masks up ...67 comments 20 Recs
- � � All fiction writers are welcome to post chapters and excerpts as comments in this thread. Use the header button to format/enlarge the title of your work in progress from which your material comes,10 comments 6 Recs
- Donald Maass is the founder of the Donald Maass Literary Agency. He handles hundreds of manuscripts each year and has sold more than 150 novels per year to major publishers. He’s a literary agent, ...28 comments 11 Recs
- AIPAC (the pro-Israeli lobby) may be using dark money to defeat progressive Democrat Susheela Jayapal in the Democratic primary for Oregon’s Third Congressional District (Portland area)...12 comments 13 Recs
- I am not anti-Israel; nor am I anti-Palestine. I am anti-war and pro-peace.1 comments 3 Recs
- WELCOME TO THE EVENING SHADE YOU WILL FIND in the DIARIES a LOT of POLITICS AND EVEN MORE CRITTERS THE PERSON who MAKES the FIRST COMMENT WILL GET TWO CRITTERS EVERY PERSON WHO COMMENTS WILL GET A CRITTER RULES IN THE DIARY WHEN YOU FIND SOMETHING in...213 comments 25 Recs
- O’Possom, the dangerous demagogue, had an Irish seven-course meal at the Lincoln-Reagan dinner. At the first criminal trial against a former American President, a porn star testified that the unexplained paranormal activity in her home was “debunked”...4 comments 15 Recs
- How do you alleviate a moral disaster in a fashion that is consistent with the fact that a state will, and in my judgment, should defend itself? And that we don't really like the idea of tens of thousands of civilians paying the price for that? Lawfare...7 comments 6 Recs
- Greetings folks! � Welcome to A Song of Zion, our weekly check-in and virtual minyan for Jews on Daily Kos. This is an open thread, and we treat it as a safe space for Jewish folks here. Non-Jews ...25 comments 6 Recs
- A Romance of Many Dimensions You, who are blessed with shade as well as light, you, who are gifted with two eyes, endowed with a knowledge of perspective, and charmed with the enjoyment of various ...12 comments 10 Recs
- The Republican Party, in concert with their aspiring authoritarian Dear Leader, Donald Trump, has long been fiercely opposed to the basic tenets of democracy. They impose flagrantly gerrymandered redistricting, engage in overt voter suppression...25 comments 86 Recs
- There is an update on what is now officially the Worst Job in Russia — night watchman at an oil refinery. Salavat is in Bashkortostan and is north of Orenburg, where all the bad flooding took place. The drone that carried out the strike was a converted...25 comments 210 Recs
- “And the worms / ate into his brain” line from Pink Floyd…better a dead one than half a live one. Robert F. Kennedy Jr. once admitted he was suffering mental problems owing to a worm eating part of his brain. If the brain worm is causing him to espouse...29 comments 64 Recs
- Welcome to the Street Prophets Coffee Hour cleverly hidden at the intersection of religion, art, science, food, and politics. This is an open thread where we can share our thoughts and comments ...7 comments 16 Recs
- Joe Biden shades Trump in Wisconsin, making it clear that he built it while Trump conned Americans. The price we pay for Bezos and Gates. Is America a Democracy or a Plutocracy? Joe Biden shades Trump in Wisconsin. Watch Politics Done Right T.V. here....3 comments 2 Recs
- The Tampa Bay Aquarium runs a dolphin boat twice a day for the tourists. It’s also a nice little trip around the Bay. Some photos. The “Bay Spirit II” tourist ...5 comments 11 Recs