hadoop - namespace image and edit log -

-

from book "hadoop definitive guide", under topic namenodes , datanodes mentioned that:

the namenode manages filesystem namespace. maintains filesystem tree , metadata files , directories in tree. information stored persistently on local disk in form of 2 files: namespace image , edit log.

secondary namenode, despite name not act namenode. main role periodically merge namespace image edit log prevent edit log becoming large.

i having confusion these files namespace , edit log.

namespace image storing metadata.

so, questions

  1. what edit log? , role?
  2. can explain statement "its main role periodically merge namespace image edit log prevent edit log becoming large."?

please can explain me edit log? role of log file?

initially when namenode first starts fsimage file empty. when ever namenode receives create/update/delete request request first recorded edits file durability once persisted in edits file in-memory update made. because read requests served in-memory snapshot of metadata.

its main role periodically merge namespace image edit log prevent edit log becoming large.

so, see edits file keeps on growing out bounds @ point. if namenode restarted or reason went down , brought up, has no memory representation of metadata so, has read edits file , rebuild snapshot in-memory, might take while based on edits file size.

as edits wal (write ahead log) events have written 1 after (append only), there no updates in file prevent random disk seeks.

to prevent overhead (or keep edits file manageable) secondarynamenode introduced. sole purpose of snn make sure edits file not grow out of bounds. so, default snn triggers process called checkpointing when ever edits file reaches 64mb or every 1 hour (which ever comes first).

checkpointing process self simple, snn tells nn role current edits log , create new edits files called edits.new, snn copies on fsimage , edits file nn , starts applying events in edits file existing fsimage file (brought nn), once completed new fsimage file sent nn , nn replaces existing fsimage new 1 sent on snn , renames edits.new edits. nn has current version of fsimage has events applied edits file.

so, if namenode restarted after checkpointing has been completed, namenode has load fsimage memory , apply recents updates edits log (which got filled after checkpoint has been completed) make sure has date view of namespace more efficient.


Comments

Post a Comment

Popular posts from this blog

c++ - QTextObjectInterface with Qml TextEdit (QQuickTextEdit) -

-
i registred handler simple qtextobjectinterface , draws 10x10 red rectangle. when used qtextedit in normal qwidget app, worked. when used qquicktextedit (textedit qml component) in qt quick app, doesn't worked (nothing drawed, rectangle in textedit reserved, because when change cursor position, notice there something, empty space, nothing drawed. qtextobjectinterface intrinsicsize method called (that explains why see there empty space 10x10), drawobject method isn't. i did research , found problem here: qquicktextedit.cpp qt 5.3.0 sources (line 1821) qsgnode *qquicktextedit::updatepaintnode(qsgnode *oldnode, updatepaintnodedata *updatepaintnodedata) { . . . if (textframe->firstposition() > textframe->lastposition() && textframe->frameformat().position() != qtextframeformat::inflow) { updatenodetransform(node, d->document->documentlayout()->frameboundingrect(textframe).topleft()); c...
Read more

javascript - angular ng-required radio button not toggling required off in firefox 33, OK in chrome -

-
i trying toggle required attribute on radio button using angular show html5 popup. i cannot following code function correctly in firefox 33. function correctly in chrome. required property being toggled via checkbox value, in firefox, when set false radio button still acts required , shows html5 required popup. see plnkr below example doesn't work me in ff 33. <!doctype html> <html> <head> <script data-require="angular.js@*" data-semver="1.3.1" src="//code.angularjs.org/1.3.1/angular.js"></script> <link href="style.css" rel="stylesheet" /> <script src="script.js"></script> </head> <body ng-app="myapp"> <script> angular.module('myapp',[]) .controller('mycontroller',['$scope',function($scope) { $scope.foorequired=true; $scope.submitform = function() { $scope.submitmessage = ...
Read more

xcode - Swift Playground - Files are not readable -

-
Image
files not readable in swift playground . how make files readable? same code runs on xcode terminal app, fails on swift playground. demo code below. import foundation println("hello, world!") var fname:string = "/users/holyfield/desktop/com.apple.iconcomposer.plist" var fm:nsfilemanager = nsfilemanager.defaultmanager() if(fm.fileexistsatpath(fname)){ println("file exists") if(fm.isreadablefileatpath(fname)){ println("file readable") var fd:nsdata? = nsdata(contentsoffile: fname) println(fd?.length) let pl = nsdictionary(contentsoffile: fname) println(pl?.count) println(pl?.allkeys) }else{ println("file not readable") } } else{ println("file not exists") } sample images: i have thank nate cook first quick response , solid answer. just case share answer post, title misleading. see nate's answer here: https://stackoverf...
Read more