urlquery.net

[tyriel]

Hi,

I've recently lunched my new project which is much like webpawt and jsunpack, it uses a different approach and gathers alot of data from what a browser does when accessing a site.
It even deobuscates Blackhole exploit kit, which webpawet struggles with. It only handles webpages and not PDF or flash files. It currently is in beta phase but should be good enough to be usefull (and hopefully stable enough).

Sample report from a maliciouse site: http://urlquery.net/report.php?id=1857

Check it out at http://urlquery.net  

Feedback is most welcome!

[SysAdMini]

Sounds interesting. I'll try it.

Welcome to MDL !

[raphael]

Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?

[tyriel]

Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?


Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version

[raphael]

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?

Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version

I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).

[SysAdMini]

Feature requests:

- referer url as an input parameter
- RSS feed of analyzed urls

[Amishrabbit]

Very interesting project. Thanks for bringing it here.

In the report.php page, under the HTTP Transactions header:

- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.
- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:

http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html

Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271

I look forward to seeing this progress.

[tyriel]

I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).

A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.

I'm not saving any .pcap file from the network traffic. I hook into the requests and responses to browser and save those. You'll loose the data from the lower levels in OSI model, but you get what the browser actual receives/handles of data. Atm i find this sufficient, having this and pcap would be alot of duplicate data, it might come it the future but I'm not sure. When downloading the data from HTTP conversations i recommend displaying those in a hex editor like the one from McAfee, FileInsight. (its free

I haven't done much work on the report page yet so it will change alot in in future. Atm most of the work has gone into the backend of the system, but I'll take your views into consideration.

- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:
http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html

Good idea, I'll have to look into how to accomplish this.

Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271

Couldn't find any easy fix for this so I'll put it on my todo list.

- referer url as an input parameter
- RSS feed of analyzed urls

RSS feeds of the latest submitted URLs are now available (and twitter)
I'm currently working on getting advanced settings and referer to work.


Thanks for the input!

[raphael]

I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).

A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

Nice, thanks!

The results of the last list in BGP Ranking: http://bgpranking.circl.lu/asns?asn=&source=URLQuery


EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.

[tyriel]

EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.

yes, just checked it

[raphael]

It is fine, the problem was on my side

And thanks again, it gives quite interesting results!

[tyriel]

Hey!

Those using urlquery.net probably have probably noticed the downtime the last weeks. I've been traveling alot lately making it hard to troubleshoot the problem, and very little time for development.  I've just now updated urlquery to my latest development branch, getting the service back online. Very sorry for the long service downtime.

The major updates are:
 - Most of the changes has been in the backend system with a better signature and detection engine in place.
 - It now also spoofs the java version making it easier to spot java exploit since exploit kits will load this code aswell. Currently the java version is hardcoded.
 - It will also create a domain access map from the HTTP requests/responses
      example -> http://urlquery.net/domainmap.php?id=4
 - Updates to the report pages

Input and thoughts are welcome.


There are more updates planned for the future
(feel free to come with suggestions)

[SysAdMini]

I have missed your service.  I'm glad that  it is back online.

[MysteryFCM]

It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87

[tyriel]

It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87

The reputation field does not include what urlquery says about it only what other external sites classifies the URL as. This was changed in the update last week, but i'll change it back if this is less intuitive.