Home› Buying & Selling Topics
Shopping at LEGO or Amazon?
Please use our links: LEGO.com • Amazon
As an Amazon Associate we earn from qualifying purchases.

Automated BrickLink cost minimization program available

12346

Comments

  • CrucioCrucio Member Posts: 2
    @Titus I guess your computer has run out of memory. I can only guess that the parts list you search for is too large or there are to many shops you are searching in.
  • CrucioCrucio Member Posts: 2
    I tried to run it out of the box, with the needed currency change do GErmany as in the readme described, i get a massive bunch of:
    "warning: load: can not read non-ASCII portions of UTF characters; replacing unreadable characters with '?'" Errors
    and also an:
    ERROR during Price Guide parsing, likely due to currency mismatch.
    Non-US users, see README notes for instructions on making modifications.
    >>>error: Stopping.

    Anybody any ideas?
  • andystarandystar Member Posts: 275
    @Titus This happens frequently when the number of vendor combinations and solutions gets large, for my machine this occurs at 4 vendor point when the # of vendor combo solutions exceeds ~50k. Typically, in my experience, this can be resolved by turning off the use of the compiled code functions, but expect a significantly longer runtime. I assume the compiled code is written for execution speed and thus consumes a lot of memory whereas the matlab code bt @atxdad is written to keep memory usage in check.
  • paynemicpaynemic Member Posts: 6
    HI all,
    First post. LOVE this concept. As an engineer, this was the first thing I thought of when I discovered bricklink. Great job.

    On my first trial run of the scripts using octave and guioctave, it runs to the question of whether or not to use PAB as a vendor. I said yes and got an error and it stopped running. I reran and said no to PAB and got what seemed like the same error. Here it is:

    Removed 178 vendors for which less expensive alternatives exist.
    Inventories of 364 unique stores saved to: Merged price guide data.mat
    warning: C:\Octave\scripts\find_minimum_cost.m: possible Matlab-style short-circuit operator at line 172, column 14
    warning: C:\Octave\scripts\find_minimum_cost.m: possible Matlab-style short-circuit operator at line 260, column 13
    warning: C:\Octave\scripts\find_minimum_cost.m: possible Matlab-style short-circuit operator at line 278, column 13
    warning: C:\Octave\scripts\find_minimum_cost.m: possible Matlab-style short-circuit operator at line 470, column 13
    warning: C:\Octave\scripts\find_minimum_cost.m: possible Matlab-style short-circuit operator at line 512, column 13
    error: caught execution error in library function
    error: called from:
    error: C:\Octave\scripts\find_minimum_cost.m at line 107, column 22
    error: C:\Octave\scripts\RUN_ALL.m at line 15, column 1
    >>

    Any thoughts? BTW, I'm a FORMER mechaincal engineer turned dentist so I am super rusty with this sort of thing...

    Thanks for any help in advance,
    paynemic
  • paynemicpaynemic Member Posts: 6
    After re-reading my post, I think more data could be useful. My run was straight RUN_ALL with no modifications (as stated in the readme--to check the plumbing). Running the Landspeeder demo data.
  • paynemicpaynemic Member Posts: 6
    Sorry for the crazy posting... Here is the output of the third time I ran it, again selecting Y when asked about PAB:

    Would you like to include LEGO Pick-A-Brick as a vendor? Y/[N]: y
    Reading PAB inventory from PAB_inventory_US_Sep_28_2012.bsx
    error: invalid concatenation of cell array with matrix
    error: called from:
    error: C:\Octave\scripts\merge_price_guide_data.m at line 113, column 38
    error: C:\Octave\scripts\RUN_ALL.m at line 14, column 1
    >>
  • TitusTitus Member Posts: 79
    Thanks andystar. My confusion was because I wasn't checking that many parts (I've checked more with no problems before) and I was also utilizing a pruned list that seemed to only include ~300 vendors. But, I guess if every vendor had some large number of parts I was looking for, maybe that caused the # of vendor solution combos to skyrocket.
    andystar said:

    @Titus This happens frequently when the number of vendor combinations and solutions gets large, for my machine this occurs at 4 vendor point when the # of vendor combo solutions exceeds ~50k. Typically, in my experience, this can be resolved by turning off the use of the compiled code functions, but expect a significantly longer runtime. I assume the compiled code is written for execution speed and thus consumes a lot of memory whereas the matlab code bt @atxdad is written to keep memory usage in check.

  • atxdadatxdad Member Posts: 68
    Hi all, I'm catching up on several recently posted questions here; hopefully I address them all. First though a reminder that queries regarding error messages are best sent directly to me instead of the forum. Debugging is sometimes an iterative process over a few exchanges if it's an error I haven't seen before.

    Check the Troubleshooting section of the README for an explanation of how memory might be exceeded; vendor pruning is the recommended solution, not a reduction in your parts list.

    The same section also mentions that Octave 'warnings' are benign, and an often overlooked step in the instructions for foreign usage is to clear the 'USonly' flag. If that flag has already been cleared, try the other currency prefix option mentioned for European union.

    The error that occurs specifically on line 107 of 'find_minimum_cost.m' is now known to be a problem with Octave's 'exist' command. It's broken in certain versions of Octave and is a known bug discussed on the Octave forums. Most common solution seems to be using a different version of Octave and not running Octave through a linux emulator. Now that I've seen this error in Octave a few times I should probably add it to the Troubleshooting section of the notes...

    @paynemic I have an idea why you might be seeing that; will PM you with questions to try to debug further.

    Poochy
  • paynemicpaynemic Member Posts: 6
    TO all interested: With help from atxdad, I got my problems fixed by downloading the latest release of Octave. Amazing programming! Thanks again!
  • mdoupemdoupe Member Posts: 1
    Hi all.

    I just installed and ran these scripts and I have to say they are pretty amazing. Kudos to @atxdad

    I am from Canada and I actually prefer to buy from the US (cheaper shipping most of the time). I was a little dismayed that I couldn't use the US only options and had to enable international stores.

    I did find a solution, though. You can google "US proxy servers" to find a current list of proxy servers located in the US. I am using octave, but I imagine MATLAB must have something similar. if you set the environment variable "HTTP_proxy" to the ip address and port of an active proxy server, bricklink will serve you up pages in USD and the cost minimization scripts will have no problem interpreting the results.

    for those less tech savvy: HTTP_proxy should be an IP address followed by a port number in the format:

    ipaddress:port

    example:

    123.456.789.012:8080

    where 123.456.789.012 is the example IP address and 8080 is the example port.

    you should be able to google " Proxy server", but the first hit for the US was http://www.proxynova.com/proxy-server-list/country-us/ when I searched.

    Good Luck!
  • LegobrandonCPLegobrandonCP Member Posts: 1,917
    @atxdad How do I get the liboctinterp.dll.a file that is missing and is preventing me from using the program?
  • atxdadatxdad Member Posts: 68
    @LegobrandonCP That sounds like a problem during Octave installation as it is not part of my distribution package. I don't use Octave myself, but maybe another forum user can send you suggestions.
  • Tevans333Tevans333 Member Posts: 152
    @legobrandonCP
    I had issues with the version you are using also.
    Try downloading this version instead. I am using it with no issues.

    http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.6.1 for Windows MinGW installer/

    Click on link for Octave3.6.1_gcc4.6.2_20120303.7z
    LegobrandonCP
  • LegobrandonCPLegobrandonCP Member Posts: 1,917
    Thanks @Tevans333
  • paynemicpaynemic Member Posts: 6
    Running well at this point. Tried a 5 vendor search of my entire UCS Falcon (with most rare pieces removed). It ran all night on 3 of four cores and came up with no solution. I'd like to expand the search, but it's becoming time prohibitive. So, I'm beginning the process of separating my list and using whitelist/blacklists and vendor pruning. I'm clear on blacklisting. I need to understand whitelisting better. Does that mean that I have picked one or more vendors that MUST be included in the final solution? Or does it make sure it is in the search? Wouldn't it be included in the search by default?

    Also, I'm opening up some of the interior (hidden, structural) parts to other colors. Does that increase or decrease runtime? Does that just make one part turn into multiple parts to search for in different colors? Anyway, I'm getting more and more excited.
    Thanks all!
  • atxdadatxdad Member Posts: 68
    Vendors on your whitelist are guaranteed to be included in the vendor search space regardless of any other pruning logic you apply, even if that logic would normally exclude them.

    Regarding colors, per the README:
    If left undefined, the program will download data from several popular candidate colors, if they exist, and later merge that pricing data into a single lot with the color field named 'Any Color'.

    The net effect will likely be to increase the size of your vendor search space.
  • wilfmannionwilfmannion Member Posts: 2
    Where can I get the scripts for this? I've got Octave installed but can't see a link anywhere to the actual scripts.
    Thanks
    Dave
  • patrickbranspatrickbrans Member Posts: 21
    Send your email address to @atxdad and he'll hook you up.
  • mai_webmai_web Member Posts: 1
    edited March 2013
    the script is awsome !!!!

    if only I had known of the existence for my last order
    i spent 60 euro less (total amount 180 euro) for my last shopping !!

    some tips for better implementing operability:

    on finding solution:

    add color spec. for minimizing buying cost
    into WANTED list generation

    with no solution found with example 2 vendors
    display finded minimizing solution with 2 vendors

    98% of lots -> Cost -> WANTED list vendors 1,2
    97% of lots -> Cost -> WANTED list vendors 1,2



    with no solution found with example 3 vendors
    display finded minimizing solution with 3 vendors

    96% of lots -> Cost -> WANTED list vendors 1,2,3
    94% of lots -> Cost -> WANTED list vendors 1,2,3


    great job !!!!
  • paynemicpaynemic Member Posts: 6
    So, I'm trying to cost the last 14 lots for my UCS falcon (what a beast!!). I have a valid 5 vendor solution provided by the scripts. I am trying to get a 6 vendor solution to see if cost goes down significantly. Each time I do, I get the following errors and am wondering what I am doing wrong. I have done significant pairing to redule the vendor list. I am running with 3 of my 4 cores (gotta work some...) and its the third window of octave that always barfs. That instancew of Octave seems to have to do the lions share of the work (judging by the index numbers and time to process.

    Any ideas?
    Thanks


    Index 310: 258 of 263, 98.10%
    Index 311: 259 of 263, 98.48%
    Index 312: 260 of 263, 98.86%
    Index 313: 261 of 263, 99.24%
    Index 314: 262 of 263, 99.62%
    Index 315: 263 of 263, 100.00%
    Search for vendor combinations completed in 101.7 minutes
    Search complete. 1.39909e+006 valid vendor combinations found.
    Costing 1.39909e+006 vendor groupings...
    error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt
    >>
  • atxdadatxdad Member Posts: 68
    @paynemic The program has exceeded the memory of your machine while calculating prices for 1.4 million solutions. I know the costing step could be made more memory efficient by merging it into the search step, but that would require a significant rewrite to move it.
  • DaddyDeuceDaddyDeuce Member Posts: 272
    Out of curiosity, how can I verify I have the latest version of the scripts?
  • atxdadatxdad Member Posts: 68
    After each version update I mail out the latest package to all on the distribution list; no need to ask for updates. I haven't done so in a while, so version 9 is still the latest. I may make small changes to documentation, etc. for the current version, but nothing that affects its operation.
  • FenrisAkashiFenrisAkashi Member Posts: 242
    atxdad, Thanks for doing this. I had Pm'ed you once to request the software but had gotten no response.
  • atxdadatxdad Member Posts: 68
    Hi all, It looks like after March 12, 2013, BrickSet either stopped sending me notifications of PM's, or some spam filter is blocking them without a trace. I'll catch up with all requests tonight. Sorry for the delay.
  • tomotomo Member Posts: 5
    Hi. How can I restrict the search to vendors from EU or Germany, only?
    The Shippingcosts from US to Germany are far to expensive so I will not order from there.
  • EdmanZAEdmanZA Member Posts: 5
    Is it possible to incorporate store feedback data into the script? For example, I'd like to be able to filter out vendors with total feedback below a certain threshold.
  • atxdadatxdad Member Posts: 68
    Answering some questions: There is currently no way for non-US users to limit a search by country. If a few US stores consistently appear in your search results, you can blacklist them and repeat your search with them excluded. The script does not pull feedback data for vendors, so filtering by feedback is not implemented. Because multiple solutions are provided with each run, suggest checking all solutions for vendor combinations with satisfactory feedback, or as before, blacklist any undesired vendors that consistently appear in your solutions.
  • EdmanZAEdmanZA Member Posts: 5
    Wouldn't it be possible to use the USOnly flag to filter out non-Euro stores (for users in countries that use Euro as a currency)?

    If I read the code right, it decides that a store is from the US simply by seeing if the store currency matches the home currency e.g. non-US stores have '~' before the price.
    The same logic could be used for any non-local currency.
  • tomotomo Member Posts: 5
    edited April 2013
    If the script uses the search instead of the price guide to get the list of vendors, one could use the search filter to get only vendors from the country or region you prefer.

    http://www.bricklink.com/search.asp?pg=1&colorID=11&qMin=20&itemID=264&sellerLoc=R&viewFrom=sf&regionID=-1&sz=500&searchSort=P

    The result is limited to 500 lots, but I think that's not such a big problem.
  • atxdadatxdad Member Posts: 68
    While it's still true that the script can't filter on specific country names, the 'US-only' flag is more accurately described as a check if your local currency is used by a vendor, excluding those stores that have to pass through the currency conversion. You can give it a try, but I think you'll still have to modify the currency prefix code if using non-USdollar currency.
  • tomotomo Member Posts: 5
    EdmanZA said:

    If I read the code right, it decides that a store is from the US simply by seeing if the store currency matches the home currency e.g. non-US stores have '~' before the price.
    The same logic could be used for any non-local currency.

    That would not work.
    If I'm not logged in at bricklink I see all prices in USD - so does the script. After the login all prices are converted into EUR to me.
  • tomotomo Member Posts: 5
    edited April 2013
    I've played around with the search a bit. You can use the search also without knowing the bricklink-item ID by using the query-field like this:

    http://www.bricklink.com/search.asp?q=3001&itemType=P&pg=1&colorID=11&qMin=20&sellerLoc=R&shipCountryID=DE&regionID=-1&sz=500&searchSort=P

    @atxdad: Do you think you can modify the readpriceguide.m script to use the search instead of the price guide page?
  • atxdadatxdad Member Posts: 68
    @tomo I considered that and other methods early in the development but went with parsing a single Price Guide page because I can get all data with one query. My URL skills via Matlab are minimal, and I didn't want to have to build queries and parse multiple web pages.
  • EdmanZAEdmanZA Member Posts: 5
    tomo said:

    That would not work.
    If I'm not logged in at bricklink I see all prices in USD - so does the script. After the login all prices are converted into EUR to me.

    It must be an IP address thing. If I'm not logged on I see all prices in my local currency (ZAR). This makes the local filter thing a bit pointless for me personally, since there's only one store selling in ZAR.
  • Bosstone100Bosstone100 Member Posts: 1,431
    Is there a reason why the scripts wouldn't ask me if I wanted to print a solution? It's bypassing it and going right to the vendor indexing prompt.

    I'm working on a project and have a solution but have no way of getting it. I haven't made any changes at all.


    Thanks,


    Mike
  • graphitegraphite Member Posts: 3,275
    ^ if this is in GUIOctave, I've had that problem before, and I think it has to do with the UI queuing up key strokes so if you accidentally hit enter twice on the previous step or at any point while processing is going on then it uses that keystroke on the next question which makes it go by it without printing it.
  • Bosstone100Bosstone100 Member Posts: 1,431
    I could really some advice on how to fix it. Thanks.
  • redheaded_jokerredheaded_joker Member Posts: 18
    I've read through the pages on this thread and I feel like I am reading a different language. Would anyone who has this program be willing to help me out? I need a lot of different pieces that I am missing from several sets, does the program work in this way? Can you enter a long list of individual items and it will find it all and put in in the cart for you?

    PM me! I would greatly appreciate for the help.
  • vonlanivonlani Member Posts: 21
    i am having some problems getting the latest script running, although i used them about half a year ago.
    i am in europe /germany so i set the USonly flag to '0' as told in the README, however i still get this error :

    ERROR during Price Guide parsing, likely due to currency mismatch.
    Non-US users, see README notes for instructions on making modifications.
    error: Stopping.
    error: called from:
    error: /Volumes/Data/lani/Downloads/BrickLink_cost_minimization_Matlab_V9b/findnextstring.m at line 12, column 5
    error: /Volumes/Data/lani/Downloads/BrickLink_cost_minimization_Matlab_V9b/parsepriceguidestring.m at line 36, column 14
    error: /Volumes/Data/lani/Downloads/BrickLink_cost_minimization_Matlab_V9b/readpriceguide.m at line 66, column 92
    error: /Volumes/Data/lani/Downloads/BrickLink_cost_minimization_Matlab_V9b/parse_wanted_list_and_price_guide.m at line 100, column 58
    error: /Volumes/Data/lani/Downloads/BrickLink_cost_minimization_Matlab_V9b/RUN_ALL.m at line 13, column 1

    i am using octave on mac os if it helps. unfortunately i have no idea how to debug a matlab script. any help would be great.
  • tomotomo Member Posts: 5
    Change the the script currency back to USD. It seems bricklink has changed something. I wrote above:
    tomo said:

    If I'm not logged in at bricklink I see all prices in USD - so does the script. After the login all prices are converted into EUR to me.

  • EdmanZAEdmanZA Member Posts: 5
    vonlani said:


    i am in europe /germany so i set the USonly flag to '0' as told in the README, however i still get this error :

    Did you also modify the code and change the currency string in parsepriceguidestring.m?

    I don't know about debugging in Octave, but you can get an idea of what data the script is retrieving from Bricklink by going to a part page in your web browser.

    The trick to this is that you first need to log out of BL and also delete some BL browser cookies (specifically isCountryID and viewCurrencyID). If you're not logged in, these two cookies are automatically set, apparently based on the location of you IP address. If you're logged in, they're set according to your BL settings.

    You'll know you've successfully killed the cookies if the page loads without the green and red blocks that indicate whether or not the store ships to you (see picture below). You'll see that if you refresh the page, the blocks will reappear.
    image

    The currency symbol shown on the page is the one that the script should search for (you can also view the page source to confirm this).


  • LeonCLeonC Member Posts: 364
    Is anyone successfully running the compiled code with Octave on 64-bit windows? I'm not able to get this working. It fails in find_minimum_cost.m at line 113 - having done a bit of digging this looks like Octave's exist() function not working correctly, or as expected. If I change it to:

    exist(mxvendorsearchloop)

    which I think is correct, then I get:

    error: could not find library or dependents: C:\Users\LEY\DOCUME~1\personal\Lego\BRICKL~1\mxvendorsearchloop.mex

    Even though the file it appears to be looking for does exist in the location it is pointing to. Does anyone have any ideas? I'm wondering if there are issues running 32-bit octave on 64-bit windows (I assume Octave is 32-bit although I'm not entirely sure how to check).

    I should note that it works fine with the uncompiled code, but I would like to get this working as I have a big-ish part list to work with.
  • LeonCLeonC Member Posts: 364
    Having done some more digging I think exist() does work correctly as I can use it at the command line to find other files (e.g. exist("RUN_ALL") returns 2).

    If I just try to run mxvendorsearchloop (which is what Ithink I was doing before by not passing the name as a string) then I get:

    error: could not find library or dependents: C:\Users\LEY\DOCUME~1\personal\Lego\BRICKL~1\mxvendorsearchloop.mex

    I get this same error if try and run any of the .mex## files.
  • atxdadatxdad Member Posts: 68
    The error in Octave's 'exist' function is a known problem, happening often enough to users that I added it to the troubleshooting section of the latest README. However, I don't know exactly what Octave version has the error and which version fixes it. If you can identify the solution, please PM me with details, and I'll add it to the documentation.
  • LeonCLeonC Member Posts: 364
    Ah, sorry. I should have rtfm. I'll take a look, thanks.
  • vonlanivonlani Member Posts: 21
    @EdmanZA : Thanks for the tip with the currency string. Apparently bricklink shows the currency based on my ip location. it doesn't matter if i am logged in or out or delete the cookies, prices are always in EUR for me.

    Changing the currency string from 'US' to 'EUR' works fine and the scripts are running now.

    as for the exits() bug in octave, i am running octave 3.6.4 compiled via fink on mac os mountain lion. maybe this helps identifying the problem.
  • MorkManMorkMan Member Posts: 916
    Why do we have to go through such hoops to get the trial version of Matlab?! Frustrating... Unless I'm doing it wrong.
  • RoscoHeadRoscoHead Member Posts: 1
    EdmanZA said:

    tomo said:

    That would not work.
    If I'm not logged in at bricklink I see all prices in USD - so does the script. After the login all prices are converted into EUR to me.

    It must be an IP address thing. If I'm not logged on I see all prices in my local currency (ZAR). This makes the local filter thing a bit pointless for me personally, since there's only one store selling in ZAR.
    I think it may be a cookie - www.bricklink.com creates a cookie "viewCurrencyID", which on my system has the value "14" (which given I'm Australian, is probably AU$). It possibly uses this setting when logged out.

    ROSCO
  • EdmanZAEdmanZA Member Posts: 5
    RoscoHead said:

    I think it may be a cookie - www.bricklink.com creates a cookie "viewCurrencyID", which on my system has the value "14" (which given I'm Australian, is probably AU$). It possibly uses this setting when logged out.

    When you access BL for the first time, so you're not logged in at all and you have no cookies, the page will choose your currency based on your IP address, according to this site http://software77.net/geo-ip/. Your country for shipping is not used. If you're accessing the page through a browser, it also sets the currency and country cookies to IP country for subsequent page loads.
    If you log in, it sets the cookies to the currency and country in your profile.
    Once cookies have been set, either to default values or profile values, every page is displayed with currency and shipping country according to the cookies.

    The problem with the script queries is that Matlab's URLread function cannot send or set any cookies, so the currency value is always the value from the IP address and the shipping country is never used.

    I've found a way to modify the Matlab URLread function to send cookies. I've sent the WIP to atxdad, so the ability to set shipping country and currency may appear at some point in a future version.
Sign In or Register to comment.

Shopping at LEGO.com or Amazon?

Please use our links: LEGO.com • Amazon

Recent discussions • Categories • Privacy Policy • Brickset.com

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Sign In Register

Categories

Brickset.com is a participant in the Amazon Services LLC Associates Program, the Amazon.com.ca, Inc. Associates Program and the Amazon EU Associates Programme, which are affiliate advertising programs designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.

As an Amazon Associate we earn from qualifying purchases.