Author Topic: urlquery.net  (Read 37830 times)

0 Members and 1 Guest are viewing this topic.

March 17, 2011, 08:37:54 pm
Read 37830 times

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
Hi,

I've recently lunched my new project which is much like webpawt and jsunpack, it uses a different approach and gathers alot of data from what a browser does when accessing a site.
It even deobuscates Blackhole exploit kit, which webpawet struggles with. It only handles webpages and not PDF or flash files. It currently is in beta phase but should be good enough to be usefull (and hopefully stable enough).

Sample report from a maliciouse site: http://urlquery.net/report.php?id=1857

Check it out at http://urlquery.net  :)

Feedback is most welcome!

March 17, 2011, 08:50:46 pm
Reply #1

SysAdMini

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 3335
Sounds interesting. I'll try it.

Welcome to MDL !
Ruining the bad guy's day

March 17, 2011, 10:57:12 pm
Reply #2

raphael

  • Newbie

  • Offline
  • *

  • 4
Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?

March 18, 2011, 12:54:34 pm
Reply #3

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
Your project is very interesting.

I am the maintainer of BGP Ranking:
- code: http://gitorious.org/bgp-ranking
- testing instance: http://bgpranking.circl.lu/ (sorry, it is quite limited, I will improve the website as soon as possible)

And would like to add your list of IPs in the sources of my project. Can you provide a link to the latest version of the list ?

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?


Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version :)

March 18, 2011, 03:41:11 pm
Reply #4

raphael

  • Newbie

  • Offline
  • *

  • 4

I currently don't have a away to get out the IP adresses from my DB. The search page atm only handles URLs. But it is possible to develop it, what do you need? any spesific format or listing. How do you want to access it?

Over the weekend I plan to implement support for the Adobe Reader plugin in the browser so you can specify which version it should use and report to javascript when it requests plugin version :)

I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).


March 18, 2011, 04:21:33 pm
Reply #5

SysAdMini

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 3335
Feature requests:

- referer url as an input parameter
- RSS feed of analyzed urls
Ruining the bad guy's day

March 18, 2011, 11:12:38 pm
Reply #6

Amishrabbit

  • Jr. Member

  • Offline
  • **

  • 10
Very interesting project. Thanks for bringing it here.

In the report.php page, under the HTTP Transactions header:

- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.
- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:

http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html

Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271

I look forward to seeing this progress.
-=A

March 19, 2011, 04:04:47 pm
Reply #7

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
Quote from: raphael
I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).
A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

Quote from: Amishrabbit
- "Requests" column is too narrow and the text doesn't wrap.
- "Respons" column is probably too wide (and you're missing an "E" from "Response")
- Are you saving off a .pcap of the conversation?
- I find the HTTP conversation stuff more useful than the DIG and WHOIS stuff. My personal preference would be to have that appear lower in the page. Others may differ.
I'm not saving any .pcap file from the network traffic. I hook into the requests and responses to browser and save those. You'll loose the data from the lower levels in OSI model, but you get what the browser actual receives/handles of data. Atm i find this sufficient, having this and pcap would be alot of duplicate data, it might come it the future but I'm not sure. When downloading the data from HTTP conversations i recommend displaying those in a hex editor like the one from McAfee, FileInsight. (its free :))

I haven't done much work on the report page yet so it will change alot in in future. Atm most of the work has gone into the backend of the system, but I'll take your views into consideration.

Quote from: Amishrabbit
- How about querying the reputation scores for domains you run queries against, using tools such as the ones listed here:
http://zeltser.com/combating-malicious-software/lookup-malicious-websites.html
Good idea, I'll have to look into how to accomplish this.


Quote from: Amishrabbit
Looks like there's a bit of work to do with busted secure connection attempts, eg. http://urlquery.net/screenshot.php?id=271
Couldn't find any easy fix for this so I'll put it on my todo list.


Quote from: SysAdMini
- referer url as an input parameter
- RSS feed of analyzed urls

RSS feeds of the latest submitted URLs are now available (and twitter) :)
I'm currently working on getting advanced settings and referer to work.


Thanks for the input! :)

March 22, 2011, 11:05:00 am
Reply #8

raphael

  • Newbie

  • Offline
  • *

  • 4
Quote from: raphael
I just need an URL like http://urlquery.net/ip.txt and ip.txt with one IP per line. And the list should be updated regularly (once a day is enough).
A list of IPs should now be available from from http://urlquery.net/ip.txt this is updated once a day (24:00 CET).
Do note that several of the IPs listed in the db are not malicious as good sites has been used for testing.

Nice, thanks!

The results of the last list in BGP Ranking: http://bgpranking.circl.lu/asns?asn=&source=URLQuery


EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.

March 27, 2011, 07:59:28 pm
Reply #9

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
Quote from: raphael
EDIT: are you sure the list is updated once a day? I had no changes since the 22.03.

yes, just checked it

March 29, 2011, 11:55:51 am
Reply #10

raphael

  • Newbie

  • Offline
  • *

  • 4
It is fine, the problem was on my side :)

And thanks again, it gives quite interesting results!

June 23, 2011, 10:33:43 pm
Reply #11

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
Hey!

Those using urlquery.net probably have probably noticed the downtime the last weeks. I've been traveling alot lately making it hard to troubleshoot the problem, and very little time for development.  I've just now updated urlquery to my latest development branch, getting the service back online. Very sorry for the long service downtime.

The major updates are:
 - Most of the changes has been in the backend system with a better signature and detection engine in place.
 - It now also spoofs the java version making it easier to spot java exploit since exploit kits will load this code aswell. Currently the java version is hardcoded.
 - It will also create a domain access map from the HTTP requests/responses
      example -> http://urlquery.net/domainmap.php?id=4
 - Updates to the report pages

Input and thoughts are welcome.


There are more updates planned for the future :)
(feel free to come with suggestions)

June 23, 2011, 10:50:43 pm
Reply #12

SysAdMini

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 3335
I have missed your service.  I'm glad that  it is back online.
Ruining the bad guy's day

June 27, 2011, 03:58:58 pm
Reply #13

MysteryFCM

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 1693
  • Personal Text
    Phishing Phanatic
    • I.T. Mate
It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87
Regards

Steven Burn
I.T. Mate / hpHosts
it-mate.co.uk / hosts-file.net

June 28, 2011, 07:05:22 am
Reply #14

tyriel

  • Jr. Member

  • Offline
  • **

  • 14
    • urlQuery
It seems to be confused. It detected the exploits, but still says it's safe?

http://urlquery.net/report.php?id=87

The reputation field does not include what urlquery says about it only what other external sites classifies the URL as. This was changed in the update last week, but i'll change it back if this is less intuitive.