The OP said: one url per line, with everything else stripped out.
Originally Posted by KC-JRPark
grep isn't going to do that for you. Nor is your regex going to even match what the OP is looking for.
sed could perhaps get you most or all of the way but whether sed can work for you easily or not is a matter which the format of your html will decide. If every URL sits neatly on one line, then sed can work.
If URLs cross line boundaries everything gets more complicated. For an example, here's a complex sed "script" (which proves sed isn't the right tool for dealing with complex parsing of HTML):
Myself, unless the need is dirt simple, I will tend to reach for a programming language (Python first in my case) that supports extended regular expressions and the "not greedy" operator.
ahref_re = re.compile(r'<a href="(http(?:s|)://.*?)".*?>(.*?)</a>', re.I|re.M)
# lets parse a local copy of the "A List Apart" home page http://www.alistapart.com/
example_html = open('example.html').read()
return [link for link in ahref_re.findall(text)
if 'img src' not in link]
if __name__ == '__main__':
for link, link_text in simple_a_extractor(example_html):
print("%-65s %s" % (link, link_text))
Maybe that's enough. If not - if the HTML is truly awful/complex (often one and the same) then we'd need to feed the HTML to a (X)HTML parser to normalize it and then use the parser to ship us the anchor links. That's beyond the scope of this discussion (so far).
http://www.alistapart.com/issues/289 No. <em>289</em>
http://www.alistapart.com/articles/erskine-design-redesign/ Erskine Design Redesign
http://www.alistapart.com/authors/c/simoncollison Simon Collison
http://www.alistapart.com/articles/redesigning-your-own-site/ Redesigning Your Own Site
http://www.alistapart.com/authors/a/lealea Lea Alcantara
http://www.alistapart.com/articles/throughthestorm/ Designing Through the Storm
http://www.alistapart.com/authors/s/walterstevenson Walter Stevenson
http://www.coudal.com/deck/ Ad via The Deck
“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”