Malware Domain List

Malware Related => Malware Analysis => Topic started by: tyriel on March 17, 2011, 08:37:54 pm

Title: urlquery.net
Post by: tyriel on March 17, 2011, 08:37:54 pm
Hi,

I've recently lunched my new project which is much like webpawt and jsunpack, it uses a different approach and gathers alot of data from what a browser does when accessing a site.
It even deobuscates Blackhole exploit kit, which webpawet struggles with. It only handles webpages and not PDF or flash files. It currently is in beta phase but should be good enough to be usefull (and hopefully stable enough).

Sample report from a maliciouse site: http://urlquery.net/report.php?id=1857

Check it out at http://urlquery.net  :)

Feedback is most welcome!
Title: Re: urlquery.net
Post by: SysAdMini on March 17, 2011, 08:50:46 pm
Sounds interesting. I'll try it.

Welcome to MDL !
Title: Re: urlquery.net
Post by: raphael on March 17, 2011, 10:57:12 pm
Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?
Title: Re: urlquery.net
Post by: tyriel on March 18, 2011, 12:54:34 pm
Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?


Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version :)
Title: Re: urlquery.net
Post by: raphael on March 18, 2011, 03:41:11 pm

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?

Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version :)

I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).

Title: Re: urlquery.net
Post by: SysAdMini on March 18, 2011, 04:21:33 pm
Feature requests:

- referer url as an input parameter
- RSS feed of analyzed urls
Title: Re: urlquery.net
Post by: Amishrabbit on March 18, 2011, 11:12:38 pm
Very interesting project. Thanks for bringing it here.

In the report.php page, under the HTTP Transactions header:

- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.
- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:

http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html

Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271

I look forward to seeing this progress.
Title: Re: urlquery.net
Post by: tyriel on March 19, 2011, 04:04:47 pm
Quote from: raphael
I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).
A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

Quote from: Amishrabbit
- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.
I'm not saving any .pcap file from the network traffic. I hook into the requests and responses to browser and save those. You'll loose the data from the lower levels in OSI model, but you get what the browser actual receives/handles of data. Atm i find this sufficient, having this and pcap would be alot of duplicate data, it might come it the future but I'm not sure. When downloading the data from HTTP conversations i recommend displaying those in a hex editor like the one from McAfee, FileInsight. (its free :))

I haven't done much work on the report page yet so it will change alot in in future. Atm most of the work has gone into the backend of the system, but I'll take your views into consideration.

Quote from: Amishrabbit
- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:
http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html
Good idea, I'll have to look into how to accomplish this.


Quote from: Amishrabbit
Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271
Couldn't find any easy fix for this so I'll put it on my todo list.


Quote from: SysAdMini
- referer url as an input parameter
- RSS feed of analyzed urls

RSS feeds of the latest submitted URLs are now available (and twitter) :)
I'm currently working on getting advanced settings and referer to work.


Thanks for the input! :)
Title: Re: urlquery.net
Post by: raphael on March 22, 2011, 11:05:00 am
Quote from: raphael
I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).
A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

Nice, thanks!

The results of the last list in BGP Ranking: http://bgpranking.circl.lu/asns?asn=&source=URLQuery


EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.
Title: Re: urlquery.net
Post by: tyriel on March 27, 2011, 07:59:28 pm
Quote from: raphael
EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.

yes, just checked it
Title: Re: urlquery.net
Post by: raphael on March 29, 2011, 11:55:51 am
It is fine, the problem was on my side :)

And thanks again, it gives quite interesting results!
Title: Re: urlquery.net
Post by: tyriel on June 23, 2011, 10:33:43 pm
Hey!

Those using urlquery.net probably have probably noticed the downtime the last weeks. I've been traveling alot lately making it hard to troubleshoot the problem, and very little time for development.  I've just now updated urlquery to my latest development branch, getting the service back online. Very sorry for the long service downtime.

The major updates are:
 - Most of the changes has been in the backend system with a better signature and detection engine in place.
 - It now also spoofs the java version making it easier to spot java exploit since exploit kits will load this code aswell. Currently the java version is hardcoded.
 - It will also create a domain access map from the HTTP requests/responses
      example -> http://urlquery.net/domainmap.php?id=4
 - Updates to the report pages

Input and thoughts are welcome.


There are more updates planned for the future :)
(feel free to come with suggestions)
Title: Re: urlquery.net
Post by: SysAdMini on June 23, 2011, 10:50:43 pm
I have missed your service.  I'm glad that  it is back online.
Title: Re: urlquery.net
Post by: MysteryFCM on June 27, 2011, 03:58:58 pm
It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87
Title: Re: urlquery.net
Post by: tyriel on June 28, 2011, 07:05:22 am
It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87

The reputation field does not include what urlquery says about it only what other external sites classifies the URL as. This was changed in the update last week, but i'll change it back if this is less intuitive.


Title: Re: urlquery.net
Post by: MysteryFCM on June 28, 2011, 01:42:00 pm
Ah, cheers :)

May be an idea to make that a little clearer, yes.
Title: Re: urlquery.net
Post by: SysAdMini on June 28, 2011, 02:11:29 pm
I would like to see a column "Host" in http request table. I don't like that I have to click on each request line to see the host.
You could make the "Reponse" column smaller, but add a host column.
Title: Re: urlquery.net
Post by: tyriel on June 28, 2011, 02:29:47 pm
I would like to see a column "Host" in http request table. I don't like that I have to click on each request line to see the host.
You could make the "Reponse" column smaller, but add a host column.

I can add the "Host" row from the http request header to the default text before you expand it. Sounds ok?
Title: Re: urlquery.net
Post by: SysAdMini on June 28, 2011, 02:39:08 pm
I would like to see a column "Host" in http request table. I don't like that I have to click on each request line to see the host.
You could make the "Reponse" column smaller, but add a host column.

I'll can add the "Host" row from the http request header to the default text before you expand it. Sounds ok?

Sounds good.
Title: Re: urlquery.net
Post by: SysAdMini on September 08, 2011, 04:48:15 pm
Let's start with reporting about missing detections.

Incognito exploit kit
example
Code: [Select]
buyaion.cu.cc/showthread.php?t=82651514
New Blackhole kit version
Code: [Select]
dreth543rwfdegrhjt.cz.cc/t/b56696ed19ad9fdfd35260d0a21bf00f
Title: Re: urlquery.net
Post by: SysAdMini on September 09, 2011, 07:00:50 am
No detection for exploits of CrimePack

Code: [Select]
greatyoutubevideos.info/nolock/index.php
vb6protected.com/nolock/index.php
Title: Re: urlquery.net
Post by: tyriel on September 12, 2011, 03:18:24 pm
No detection for exploits of CrimePack

Code: [Select]
greatyoutubevideos.info/nolock/index.php
vb6protected.com/nolock/index.php


I'll have a closer look at those URL, not sure if they contain CrimePack tho, as one seems to use some Java code and the other seems to be dead at time of visit.

I'll update the BlackHole and Incognito signatures tonight with new patterns.


Thanks for feedback MDL! :)

Title: Re: urlquery.net
Post by: tyriel on September 13, 2011, 04:50:37 pm
Let's start with reporting about missing detections.

Incognito exploit kit
example
Code: [Select]
buyaion.cu.cc/showthread.php?t=82651514


Anyone know what version of incognito this is?

I remember the old format from v2.0 was:

Code: [Select]
/in.php?a=QQkFBwQHBAEABQQMEkcJBQcEBwYABQcHDA==